Context-aware real-time meeting audio transcription
The context-aware transcription system addresses transcription challenges in conferencing by using a meeting-specific model to enhance accuracy and reduce resource consumption, benefiting hearing-impaired and multilingual participants.
Patent Information
- Application Number
- CN202080028127.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-12
- Filing Date
- 2020-03-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-03-23
AI Technical Summary
Conventional conferencing technologies struggle with transcription quality for individuals with hearing impairments and language barriers, leading to inefficient use of computing resources due to poor transcription accuracy and the need for re-transcription.
A context-aware real-time transcription system generates high-quality transcriptions using a meeting-specific statistical language model, incorporating domain-specific terminology and updating in real-time with attendee data, to improve accuracy and reduce resource consumption.
Enhances participation for hearing-impaired and multilingual attendees by providing accurate real-time transcriptions, reducing the need for editing and conserving computing resources.
Smart Images

Figure CN113678153B_ABST
Abstract
Description
Background Art
[0001] Groups of individuals typically meet using various types of audio and audio-visual ("A / V") conferencing technologies. In meetings such as these, each meeting attendee utilizes a computing device configured with a conference client application and appropriate hardware to generate audio and / or video of the attendee. During the meeting, the audio and / or video generated by each attendee computing device is shared with the computing devices of the other attendees. In this manner, each of the attendees is able to hear and / or see the other attendees in the meeting.
[0002] While conferences such as those described above can be very useful for many types of users, some conferencing technologies make these kinds of conferences less beneficial for other types of participants. For example, people with hearing impairments may have difficulty participating in such conferences or may not be able to participate at all. Participants in a conference who do not speak the same language as the other participants may also be unable to communicate effectively or at all using current conferencing technologies.
[0003] Technologies exist for transcribing audio; however, these technologies suffer from several technical limitations. For example, many transcription solutions only perform offline transcription of the audio and, therefore, are not useful to attendees during a meeting. Furthermore, the quality of the transcriptions generated by current technologies can be poor. This can occur, for example, if attendees use jargon, abbreviations, or other types of domain-specific terminology. This can also occur when attendees in a meeting speak different languages.
[0004] Poor quality transcriptions can also result in inefficient use of computing resources such as memory, processing cycles, storage, and power. For example, a low quality transcription may need to be re-transcribed or edited to correct errors. This process will unnecessarily consume computing resources such as those identified above.
[0005] It is with respect to these and other technical challenges that the disclosures made herein are presented. Summary of the invention
[0006] Techniques for context-aware real-time conference audio transcription are disclosed. By implementing the disclosed techniques, high-quality transcription of audio generated by conference participants in a conference can be performed in real-time during the conference and displayed to conference participants. By generating a conference-specific statistical language model and using the model to perform the transcription, the quality of the transcription can be improved compared to previous solutions.
[0007] By generating and presenting high-quality transcriptions of conference audio in real time during a meeting, conference attendees with hearing impairments can participate in the meeting more effectively. In addition, by converting transcriptions generated using the techniques disclosed herein, speakers of different foreign languages can participate in the meeting more effectively. Finally, computing resources, such as those described above, can be saved by providing high-quality real-time conference transcriptions that do not need to be edited. Other technical benefits not specifically mentioned herein can also be achieved by implementing the disclosed subject matter.
[0008] In order to achieve the technical benefits briefly mentioned above, a context-aware transcription system is provided. The context-aware transcription system can use a meeting-specific statistical language model to generate high-quality transcriptions of audio generated during a meeting. Specifically, the context-aware transcription system can include a language model preparation service that retrieves meeting-specific data before or during a meeting. The meeting-specific data can include, but is not limited to: messages generated by or otherwise associated with a meeting attendee, attachments to messages, names of meeting attendees, or files created by or otherwise associated with a meeting attendee. The meeting-specific data can include jargon, abbreviations, or other types of domain-specific terms associated with a meeting attendee.
[0009] Once the language model preparation service has collected the meeting-specific data, the service utilizes the meeting-specific data to generate a meeting-specific statistical language model. In some configurations, the meeting-specific statistical language model is stored during the duration of the meeting for recovery purposes and is destroyed after the end of the meeting.
[0010] The speech transcription service can utilize the meeting-specific statistical language model to generate a text transcription of the audio of the meeting. In some configurations, the speech transcription service utilizes the meeting-specific statistical language model in conjunction with a base (ie, non-domain-specific) language model to generate the transcription.
[0011] In some configurations, the transcription is transmitted to a computing device associated with a conference attendee during the conference. The computing device is configured to present the transcription in a user interface ("UI") during the conference in real time or near real time. For example, the UI can overlay the transcribed text on video content generated by the attendee during the conference.
[0012] In some configurations, the language model preparation service generates a meeting-specific statistical language model in response to receiving a pre-meeting signal generated by a scheduling processor and triggering service (“STS”). The pre-meeting signal can include data describing the meeting, such as, for example, data identifying the meeting and data identifying a start time for the meeting.
[0013] To generate the pre-meeting signal, the STS can retrieve a list of upcoming meetings on a predefined schedule (e.g., once per day). Using this data and the start time of the meeting, the STS can generate the pre-meeting signal a predetermined amount of time (e.g., one hour) before the start time of the meeting. In this way, the generation of a meeting-specific statistical language model for the meeting can be initiated at a predetermined amount of time before the meeting.
[0014] The STS may also or alternatively subscribe to receive conference change event notifications describing changes to the conference, such as, for example, adding or removing attendees, adding or removing files, or modifying the start time of the conference. The STS can generate a pre-conference signal in response to receiving a conference change event notification. This can occur before or during the conference. For example, when a participant joins or leaves a conference, an additional signal can be generated during the conference. In this way, the conference-specific statistical language model for the conference can be updated with additional or modified conference-specific data before or during the conference to keep the context for the transcription focused on the active participants and further improve the accuracy of the generated transcription.
[0015] It should be appreciated that the subject matter described above can be implemented as a computer-controlled apparatus, a computer-implemented method, a computing device, or an article such as a computer-readable medium. These and various other features will be apparent by reading the following detailed description and examining the associated drawings.
[0016] This Summary section is provided to present a brief summary of some aspects of the disclosed technology in a simplified form, which are further described below in the Detailed Description. This Summary section is not intended to identify key features or essential features of the claimed subject matter, nor is it intended that this Summary be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that overcome any or all of the deficiencies noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a computing architecture diagram illustrating aspects of the configuration and operation of a context-aware transcription system according to one embodiment disclosed herein;
[0018] Figure 2 is a diagram of a computing architecture according to an embodiment disclosed herein, which shows Figure 1 Aspects of the configuration and operation of the context-aware transcription system shown in for generating a conference-specific statistical language model;
[0019] Figure 3is a diagram of a computing architecture according to an embodiment disclosed herein, which shows Figure 1 and Figure 2 Aspects of the configuration and operation of a context-aware transcription system as shown in for generating a pre-meeting signal used to trigger the generation of a meeting-specific statistical language model;
[0020] Figure 4 is a data structure diagram illustrating aspects of the configuration of a pre-meeting signal used to trigger the generation of a meeting-specific statistical language model according to one embodiment disclosed herein;
[0021] Figure 5 is a diagram of a computing architecture according to an embodiment disclosed herein, which shows Figure 1-Figure 3 Additional aspects of the configuration and operation of the context-aware transcription system shown in for performing real-time meeting transcription using a meeting-specific statistical language model;
[0022] Figure 6 is a user interface diagram according to an embodiment disclosed herein, which shows the use of Figure 1-Figure 5 The context-aware transcription system shown in presents a user interface for a transcription of the audio of a meeting;
[0023] Figure 7 is a flowchart showing a routine according to one embodiment disclosed herein, the routine illustrating reference Figure 1-Figure 6 Aspects of the operation of the described context-aware transcription system for generating a transcription of the audio of a conference in real time;
[0024] Figure 8 is a computer architecture diagram illustrating an exemplary computer hardware and software architecture for a computing device capable of implementing various aspects of the techniques presented herein; and
[0025] Fig. 9 is a network diagram illustrating a distributed computing environment in which aspects of the disclosed technology can be implemented. DETAILED DESCRIPTION
[0026] The following detailed description relates to technology for context-aware real-time conference audio transcription. As briefly discussed above, implementation of the disclosed technology can generate high-quality transcriptions of audio generated by attendees in a meeting in real time. By generating and presenting high-quality transcriptions of conference audio in real time during a meeting, conference attendees with hearing impairments can participate in the meeting more effectively. In addition, computing resources can be saved by providing high-quality real-time conference transcriptions that do not need to be edited. Other technical removals not specifically mentioned herein can also be achieved through implementation of the disclosed subject matter.
[0027] Although the subject matter described herein is presented in the general context of a network service configured for context-aware real-time audio transcription, those skilled in the art will recognize that other implementations can be performed in conjunction with other types of computing systems and modules. For example, the disclosed techniques can be used for live subtitle shorthand and translation. Those skilled in the art will also recognize that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, computing or processing systems embedded in devices (such as wearable computing devices, automobiles, home automation, etc.), minicomputers, mainframe computers, etc.
[0028] In the following detailed description, reference will be made to the accompanying drawings which form an integral part thereof and which show by way of illustration specific configurations or examples. Referring now to the drawings, in which like reference numerals represent like elements throughout the several views, various aspects of the configuration and operation of a context-aware transcription system will be described.
[0029] Figure 1 is a computing architecture diagram illustrating various aspects of the configuration and operation of the context-aware transcription system 102 according to one embodiment disclosed herein. Figure 2-Figure 7 As described in more detail, the context-aware transcription system 102 is able to use a meeting-specific statistical language model to generate high-quality transcriptions of audio generated during a meeting in real-time.
[0030] In a particular embodiment, groups of individuals can conduct a conference using associated computing devices and appropriate hardware and software. Specifically, attendees 104A and 104B in a conference (which may be collectively referred to herein as "attendees 104" or in the singular) can utilize associated computing devices 106A and 106B, respectively (which may be collectively referred to herein as "computing devices 106" or in the singular).
[0031] Computing device 106 is configured with a conference client application 108 and appropriate hardware (e.g., a microphone or webcam) for generating audio 114A and 114B and / or video, respectively, of conference attendees 104. Conference client application 108 may be: SKYPE, SKYPE FOR BUSINESS, or TEAMS conference client applications from MICROSOFT CORP., a ZOOM conference client application from ZOOM VIDEO COMMUNICATIONS, a CHIME conference client application from AMAZON.COM, or another application from another vendor that facilitates audio and / or video conferencing.
[0032] During the conference, the conference client application 108 shares the audio 114 and / or video generated by each participant's computing device 106 with the computing devices 106 of the other participants 104. In this way, each participant 104 of the participants 104 can hear and / or see the other participants 104 in the conference. Figure 1 This functionality is accomplished using networking components, server computers, and software components beyond those shown in FIG.
[0033] In some configurations, the context-aware transcription system 102 operates in conjunction with the conference client application 108 to transcribe spoken words in the audio 114 to generate a text transcript 112 of the conference. To accomplish this, the conference client application transmits the audio 114 generated by the computing device 106 to the context-aware transcription system 102. The context-aware transcription system 102 then generates a transcript 112 of the audio 114 in real time. Figure 2-Figure 7 To provide details regarding the generation of transcript 112.
[0034] In some configurations, during the meeting, the transcript 112 is transmitted to a computing device 106 associated with the meeting attendee 104. The meeting client application 108 can receive the transcript 112 and present the transcript 112 to the attendee 104 in the UI 110 in real time during the meeting. The UI 110 can, for example, overlay the text of the transcript 112 on the video content generated by the computing device 106 during the meeting. Figure 6 Detail regarding one such UI 110 is provided below.
[0035] Figure 2 is a computing architecture diagram, which shows Figure 1 , and briefly described above. Specifically, Figure 2 Various aspects of the context-aware transcription system 102 for generating a meeting-specific statistical language model 216 for transcribing the audio 114 of a meeting are illustrated.
[0036] In some configurations, a transcription policy 204 can be defined that includes data specifying whether to generate transcripts 112 for meetings scheduled by users in the organization. For example, a user 218 with appropriate administrator privileges can access an administration portal 220 using a computing device 106C and an application 202A (e.g., a web browser application).
[0037] Through the management portal 220, the user 218 can define a transcription policy 204 that indicates that the transcription 112 is to be generated. The transcription policy 204 can be stored in a substrate 206 that includes storage and processing facilities. The substrate 206 can be integrated with the context-aware transcription system 102 or implemented by another system in other configurations.
[0038] As in Figure 2 , a meeting attendee 104A or another user can utilize a computing device 106B and an application 202B (e.g., a personal information manager (“PIM”) application or another type of local or network-based calendar application) to schedule a meeting with one or more other attendees 104. Data describing the meeting is stored in a meeting definition 208, which in some configurations also resides in the substrate 206.
[0039] Meeting definition 208 can include various types of data about the meeting, including, but not limited to, a unique identifier for the meeting, the date and time of the meeting, and a list of the meeting's attendees 104. Meeting definition 208 can include other types of data not specifically identified herein.
[0040] Also as in Figure 2 As shown in , the context-aware transcription system 102 includes a language model preparation service 210 in some configurations. The language model preparation service 210 is a network service that can retrieve meeting-specific data 214 before or during a meeting. The meeting-specific data 214 can include, but is not limited to: messages (e.g., emails, text messages, instant messages, etc.) generated by or otherwise associated with a meeting attendee 104, file attachments to messages, names of meeting attendees 104, or files generated by or otherwise associated with a meeting attendee 104.
[0041] The meeting specific data 214 can be retrieved within a predefined time period. For example, messages for the previous six months (or other time period) can be retrieved for each of the meeting attendees. The meeting specific data 214 can include jargon, abbreviations, or other types of domain specific terms associated with the meeting attendees 104. The meeting specific data 214 can be identified based on the meeting definition 208 (e.g., a list of attendees) and / or other types of information.
[0042] Once the language model preparation service 210 has collected the meeting-specific data 214, the service 210 utilizes the meeting-specific data 214 to generate a meeting-specific statistical language model 216. In some configurations, the meeting-specific statistical language model 216 is stored in the substrate 206 during the duration of the meeting for recovery purposes and is deleted after the conclusion of the meeting.
[0043] The meeting-specific statistical language model 216 defines a probability distribution that can be used to predict the next word in a sequence given the words that preceded it. Such a language model learns the probabilities of word occurrences based on examples of text, such as the meeting-specific data 214. In this way, the meeting-specific statistical language model can be used to determine the probabilities of spoken words in the audio and generate a transcription based on the calculated probabilities.
[0044] The conference-specific statistical language model 216 can be implemented using a variety of techniques, including, but not limited to: an incremental language model, an n-gram language model, a uniformly distributed (zero-gram) language model, a unary language model, a bigram language model, a skip language model, a class language model, a topic language model, a neural network language model, a long short-term memory ("LSTM") model, or a recursive neural network language model.
[0045] In some configurations, the language model preparation service 210 responds to receiving a request from a scheduler and trigger service ("STS") 302 (in Figure 3 2 and generate a meeting-specific statistical language model 216 based on the pre-meeting signal 212 generated by the meeting. As will be described in more detail below, the pre-meeting signal 212 can include data describing the meeting, such as, for example, data identifying the meeting and data identifying the start time of the meeting. Figure 3 and Figure 4 Details regarding the generation of pre-meeting signal 112 are provided below.
[0046] As will also be described in more detail below, the speech transcription service 502 (in Figure 5 ) can utilize the conference-specific statistical language model 216 to generate a text transcription 112 of the audio 114 of the conference. In some configurations, the speech transcription service 502 utilizes the conference-specific statistical language model 216 in combination with a base (i.e., non-domain-specific) language model to generate the transcription 112. Figure 5 To provide details about the operation of speech transcription service 502.
[0047] Figure 3 is a diagram of a computing architecture according to an embodiment disclosed herein, which shows Figure 1 and Figure 2 1 and 2, and the context-aware transcription system 102 shown in FIG. 1 is used to generate various aspects of the configuration and operation of a pre-meeting signal 212 that triggers the generation of a meeting-specific statistical language model 216. As briefly discussed above, a network service referred to herein as a scheduling processor and triggering service ("STS") 302 generates the pre-meeting signal 212 in some configurations.
[0048] To generate the pre-meeting signal 212, the STS 302 can utilize a schedule-based process 306 and / or an event-based process 304. When utilizing schedule-based process 306, the STS 302 retrieves a list 310 of upcoming meetings on a predefined schedule (e.g., once per day). For example, but not limitation, the STS 302 might call a calendar application programming interface (“API”) 308 to retrieve a list 310 of upcoming meetings within a predetermined time period (e.g., seven days) in the future from the substrate 206 according to a predefined schedule. The list 310 of meetings can be generated based on data defining meetings 312 on a user's calendar 314 stored in the substrate 206 or in another location.
[0049] Using data identifying the start times of the meetings specified in the list 310 of upcoming meetings, the STS 302 can generate a pre-meeting signal 212 a predetermined amount of time (e.g., one hour) before the start time of each meeting. For example, but not by way of limitation, if a meeting is scheduled to start at 2:00 p.m., the STS 302 can transmit the pre-meeting signal 212 to the language model preparation service 210 for the meeting at 1:00 p.m. As briefly described above, the language model preparation service 210 generates a meeting-specific statistical language model 216 for the meeting identified by the pre-meeting signal 212. In this manner, generation of a meeting-specific statistical language model 216 for a meeting can begin at a predetermined amount of time before the start time of the meeting.
[0050] As described above, the STS may also or alternatively use event-based processing 302 to trigger the generation of pre-meeting signals 212 for a meeting. To utilize event-based processing 302, the STS 302 can subscribe to receive meeting change event notifications 316. In a particular embodiment, for example, the event signaling service 318 can generate a meeting change event notification 316 in response to detecting that a meeting has been changed. For example, the event signaling service 318 can generate a meeting change event notification 316 in response to detecting that a participant is added or removed from a meeting, files associated with a meeting are added or removed, or the start time of a meeting is modified. The meeting change event notification 316 includes data describing the detected change to the meeting.
[0051] Once the STS 302 has subscribed to receive conference change event notifications 316, the STS 302 can generate a pre-meeting signal 212 in response to receiving the conference change event notification 316. This can occur before or during the meeting. In this way, the conference-specific statistical language model 216 for the meeting can be updated with additional or modified conference-specific data 214 before or during the meeting to further improve the accuracy of the generated transcription 112 of the meeting. For example, when a notification 316 indicating that a new attendee 104 has joined the meeting is received, the conference-specific statistical language model 216 for the meeting can be updated with the conference-specific data 214 associated with the new attendee 104. After receiving a notification 316 indicating that the attendee 104 has left the meeting, the conference-specific statistical language model 216 can be updated in a similar manner. As briefly described above, both the scheduling-based processing 306 and the event-based processing 304 are used in some configurations to generate the pre-meeting signal 212.
[0052] Figure 4 is a data structure diagram illustrating aspects of the configuration of an exemplary pre-meeting signal 212 used to trigger the generation of a meeting-specific statistical language model 216 according to one embodiment disclosed herein. Figure 4 In the embodiment shown in , the pre-meeting signal 212 includes data 402A identifying the start time of the meeting, data 402B identifying a fragment (i.e., a database) where information about the meeting can be found, data 402C specifying a unique identifier for the meeting, and data 402D containing meeting-specific values.
[0053] As will be described in more detail below, language model preparation service 210 can utilize data contained in pre-meeting signal 212 to obtain meeting-specific data 214 for the meeting identified by pre-meeting signal 212. Figure 5 Additional details about the process are provided.
[0054] Figure 5 is a diagram of a computing architecture according to an embodiment disclosed herein, which shows Figure 1-Figure 3 , and as described above, for performing real-time meeting transcription using a meeting-specific statistical language model 216. As briefly described above, receipt of a pre-meeting signal 212 triggers a language model preparation service 210 to generate a meeting-specific statistical language model 216 for the meeting identified by the pre-meeting signal 212.
[0055] To generate a pre-meeting signal 212 for a meeting, a language model preparation service 212 uses the data contained in the pre-meeting signal 212 to locate and retrieve meeting-specific data 214 for the meeting from the substrate 206 and / or another location. As discussed above, the meeting-specific data 214 can include, but is not limited to: messages (e.g., emails, text messages, instant messages, etc.) generated by or otherwise associated with a meeting attendee 104, file attachments to messages, names of meeting attendees 104, or files generated by or otherwise associated with a meeting attendee 104. The meeting-specific data 214 can include jargon, abbreviations, or other types of domain-specific terminology associated with a meeting attendee 104.
[0056] Once the language model preparation service 210 has generated the meeting-specific statistical language model 216, the language model preparation service 210 can store the meeting-specific statistical language model 216 in the substrate 206 or at another location. As discussed above, the meeting-specific statistical language model 216 can be stored during the duration of the meeting for recovery purposes, and in some configurations destroyed after the end of the meeting.
[0057] As in Figure 5 , the language model preparation service 210 also provides the meeting-specific statistical language model 216 to the speech transcription service 502. As briefly described above, the speech transcription service 502 is a network service configured to receive the audio 114 of the meeting and generate a text transcription 112 of the speech contained in the audio 114. To provide this functionality, in one configuration, a multiplexer 504 is utilized to receive the audio 114 of the meeting from the participating computing devices 106. The multiplexer 504 provides the audio 114 to the speech transcription service 502. As shown in Figure 5 As shown in , the multiplexer 504 can also provide the audio 114 to each of the participating computing devices 106 .
[0058] The speech transcription service 502 receives the audio 114 and utilizes the conference-specific statistical language model 216 to generate a transcription 112 of the audio 114 in real time (i.e., as the attendee 104 speaks). The speech transcription service 502 utilizes the conference-specific statistical language model 216, in some configurations in conjunction with a base statistical language model 506. The conference-specific statistical language model 216 includes domain-specific (i.e., conference-specific) terminology, whereas the base statistical language model 506 generally does not include domain-specific terminology.
[0059] Like the conference-specific statistical language model 216, the base statistical language model 506 also defines a probability distribution that can be used to predict the next word in the sequence given the word that preceded it. The base statistical language model 506 can be implemented using various techniques, including, but not limited to: an incremental language model, an n-gram language model, a uniform distribution (zero-gram) language model, a unigram language model, a bigram language model, a skip language model, a class language model, a topic language model, a neural network language model, or a recursive neural network language model.
[0060] The speech transcription service 502 provides the transcription to the multiplexer 504 as it is generated. In turn, the multiplexer 504 provides the transcription 112 to instances of the conference client application 108 executing on computing devices 106 participating in the conference. The multiplexer 504 can also store the transcription 112 in the substrate 206 for future retrieval and viewing.
[0061] As discussed above, the conference client application 108 presents the transcript 112 in the UI 110 in real time during the conference. Figure 6 An exemplary UI 110 for presenting a transcript 112 is depicted.
[0062] Figure 6 is a user interface diagram according to an embodiment disclosed herein, which shows the use of Figure 1-Figure 5 The context-aware transcription system 102 shown in FIG. 1 presents an exemplary UI 110 of a transcription 112 of the audio 114 of a meeting. Figure 6 As shown in , the UI 110 can present the transcription 112 in real time as it is generated by the speech transcription service 502 .
[0063] Also as in Figure 6 , UI 110 can include several UI elements for controlling the presentation of transcript 112. For example, but not limitation, UI 110 can include menu 602 having UI control 604A that, when selected, turns presentation of transcript 112 in UI 110 on or off. Menu 602 can include UI controls for performing other types of functions, including, but not limited to, recording the meeting, turning video or audio on or off, displaying a chat window, and ending the meeting. Menu 602 can include other UI controls for performing other types of functions in other configurations.
[0064] A menu 606 can also be presented in the UI 110 that includes a menu item 608A that, when selected, turns the generation of the transcription 112 on or off. The menu 606 can include menu items for performing other types of functions, including, but not limited to, showing device settings, entering full screen mode, starting a video with a blurred background, showing a keypad, or turning off an incoming video. The menu 606 can include other UI controls for performing other types of functions in other configurations. In this regard, it should be appreciated that the menu items 608A and 608A may be used to display the device settings, enter full screen mode, start a video with a blurred background, show a keypad, or turn off an incoming video. Figure 6 The exemplary configuration of UI 110 shown in FIG. 1 is merely exemplary, and other types of user interfaces can be utilized to present transcription 112 in real time.
[0065] Figure 7 is a flowchart illustrating a routine 700 according to one embodiment disclosed herein, which illustrates the above reference Figure 1-Figure 6 The described context-aware transcription system 102 is used to generate various aspects of the operation of the transcription 112 of the conference audio 114 in real time. It should be appreciated that in this document, the context-aware transcription system 102 is used to generate various aspects of the operation of the transcription 112 of the conference audio 114 in real time. Figure 7 The logical operations described in the and other figures can be implemented as: (1) a sequence of computer-implemented actions or program modules running on a computing device, and / or (2) interconnected machine logic circuits or circuit modules within a computing device.
[0066] The specific implementation of the technology disclosed in this article is a matter of choice depending on the performance and other requirements of the computing device. Therefore, the logical operations described in this article are referred to as states, operations, structural devices, actions or modules in various ways. These states, operations, structural devices, actions and modules can be implemented with hardware, software, firmware, special digital logic units and any combination thereof. It should be appreciated that more or less operations can be performed than shown in the drawings and described in this article. These operations can be performed in an order different from those described in this article.
[0067] The routine 700 begins at operation 702, where the language model preparation service 210 determines whether a pre-meeting signal 212 for a meeting is received from the STS 302. As discussed above, the STS 302 can utilize the schedule-based processing 306 and the event-based processing 304 to determine to generate the pre-meeting signal 212 for the meeting. If the language model preparation service 210 receives the pre-meeting signal 212 for the meeting, the routine 700 proceeds from operation 702 to operation 704.
[0068] At operation 704, the language model preparation service 210 retrieves meeting specific data 214 for the meeting identified by the received pre-meeting signal 212. The meeting specific data 214 can be retrieved from the substrate 206 and / or from another location or locations. The routine 700 then proceeds from operation 704 to operation 706.
[0069] At operation 706, the language model preparation service 210 generates a meeting-specific statistical language model 216 using the meeting-specific data 214 retrieved at operation 704. The routine 700 then proceeds to operation 708, where the language model preparation service 210 stores the meeting-specific statistical language model 216 in the substrate 206. The routine 700 then proceeds from operation 708 to operation 710.
[0070] At operation 710, the language model preparation service 210 provides the meeting-specific statistical language model 216 to the speech transcription service 502. The routine 700 then proceeds from operation 710 to operation 712, where the speech transcription service 502 generates a transcription 112 of the audio 114 of the meeting using the meeting-specific statistical language model 216. In some configurations, the speech transcription service 502 generates the transcription 112 using the meeting-specific statistical language model 216 and the base statistical language model 506.
[0071] From operation 712, the routine 700 proceeds to operation 714, where the multiplexer 504 or another component provides the transcription 112 to the computing devices participating in the conference. In turn, the conference client application 108 executing on the computing device 106 receives the transcription 112 and presents the transcription 112 in the UI 110 described above. The routine 700 then proceeds from operation 714 to operation 716, where the multiplexer 504 stores the transcription 112 in the substrate 206 or another location for future use.
[0072] From operation 716, the routine 700 proceeds to operation 718, where it is determined whether the meeting is complete. If the meeting is not complete, the routine 700 proceeds back to operation 712, where the speech transcription service 502 continues to generate a transcription 112 of the audio 114 of the meeting in the manner described above. As also described above, the meeting-specific statistical language model 216 can be updated during the meeting based on the meeting change event notification 316 received from the event signaling service 318. If the meeting is complete, the routine 700 proceeds from operation 718 to operation 720.
[0073] At operation 720, the meeting-specific statistical language model 720 can be deleted from the substrate 206. Additionally, at operation 722, the full transcript 112 of the meeting can be made available to the meeting attendees 104. For example, the meeting client application 108 can provide functionality for enabling the attendees 104 to review the full transcript 112 of the meeting. The routine 700 then proceeds from operation 722 to operation 724, where the routine ends.
[0074] Figure 8 is a computer architecture diagram that illustrates an exemplary computer hardware and software architecture for a data processing system capable of implementing the various techniques presented herein. Figure 8 The architecture illustrated in the figure can be used to implement a server computer, a mobile phone, an e-reader, a smart phone, a desktop computer, an AR / VR device, a tablet computer, a laptop computer, or another computing device.
[0075] exist Figure 8 The data processing system 800 illustrated in FIG. 8 includes a central processing unit (“CPU”) 802, a system memory 804 including a random access memory 806 (“RAM”) and a read-only memory (“ROM”) 808, and a system bus 810 that couples the memory 804 to the CPU 802. A basic input / output system (“BIOS” or “firmware”) containing basic routines that help transfer information between elements within the data processing system 800, such as during startup, can be stored in the ROM 808. The data processing system 800 also includes a mass storage device 812 for storing an operating system 822, application programs, and other types of programs. For example, the mass storage device 812 can store the conference client application 108 when used to implement the computing device 106. The mass storage device 812 can store the speech transcription service 502 when used to implement the computing system in the context-aware transcription system 102. The mass storage device 812 can also be configured to store other types of programs and data.
[0076] Mass storage device 812 is connected to CPU 802 through a mass storage controller (not shown) connected to bus 810. Mass storage device 812 and the computer-readable media associated therewith provide non-volatile storage for data processing system 800. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk, a CD-ROM drive, a DVD-ROM drive, or a USB memory key, those skilled in the art will appreciate that computer-readable media can be any available computer storage media or communication media that can be accessed by data processing system 800.
[0077] Communication media includes computer readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier wave or other transmission mechanism), and includes any delivery medium. The term "modulated data signal" means a signal having one or more of its characteristics changed or set in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media such as a wired network or a direct wired connection and wireless media such as acoustic, radio frequency, infrared and other wireless media. Any combination of the above should also be included within the scope of computer readable media.
[0078] By way of example and not limitation, computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media include, but are not limited to: RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, Bluetooth or other optical storage devices, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by data processing system 800. For the purposes of the claims, the phrase "computer storage medium" and its various variations do not include waves or signals themselves or communication media.
[0079] According to various configurations, data processing system 800 can operate in a networked environment using logical connections to remote computers through a network such as network 820. Data processing system 800 can be connected to network 820 through a network interface unit 816 connected to bus 810. It should be appreciated that network interface unit 816 can also be utilized to connect to other types of networks and remote computer systems. Data processing system 800 can also include a computer for inputting commands from a keyboard, a mouse, a touch input, an electronic pen (in the Figure 8 The input / output controller 818 receives and processes input from a plurality of other devices (not shown) or physical sensors such as a video camera. Similarly, the input / output controller 818 can output to a display screen or other type of output device (in the Figure 8 (not shown in the figure) provides an output.
[0080] It should be appreciated that the software components described herein, when loaded into CPU 802 and run, can convert CPU 802 and overall data processing system 800 from a general purpose computing device to a dedicated computing device customized to facilitate the functions proposed herein. CPU 802 can be composed of any number of transistors or other discrete circuit elements, which can take any number of states individually or collectively. More specifically, in response to the executable instructions contained in the software modules disclosed herein, CPU 802 can operate as a finite state machine. These computer executable instructions can convert CPU 802 by specifying how CPU 802 transitions between states, thereby converting transistors or other discrete hardware elements that constitute CPU 802.
[0081] The software modules encoded in this article can also convert the physical structure of the computer-readable medium proposed in this article. In different embodiments of this specification, the specific conversion of the physical structure depends on various factors. Examples of such factors include, but are not limited to: the technology used to implement the computer-readable medium, whether the computer-readable medium is characterized as a primary or secondary storage device, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed in this article can be encoded on the computer-readable medium by converting the physical state of the semiconductor memory. For example, the software can convert the state of transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software can also convert the physical state of such components in order to store data thereon.
[0082] As another example, magnetic or optical technology can be used to implement the computer readable medium disclosed in this article. In such an embodiment, when the software is encoded therein, the software proposed in this article can convert the physical state of the magnetic or optical medium. These conversions can include changing the magnetic properties of specific locations within a given magnetic medium. These conversions can also include changing the physical characteristics or properties of specific locations within a given optical medium to change the optical properties of those locations. Other conversions of physical media are also possible without departing from the scope and spirit of this specification, and the aforementioned examples are provided only to facilitate this discussion.
[0083] In view of the above, it should be appreciated that many types of physical transformations occur in data processing system 800 in order to store and execute the software components presented herein. Figure 8The architecture shown for data processing system 800 in the embodiment of the present invention, or a similar architecture, can be used to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems, mobile devices such as smart phones, tablet computers, and AR / VR devices, as well as other types of computing devices known to those skilled in the art. It is also contemplated that data processing system 800 may not include Figure 8 All components shown in can be included in Figure 8 Other components not explicitly shown in the figure or capable of utilizing the same Figure 8 A completely different architecture is shown in .
[0084] Fig. 9 9 is a network diagram according to various embodiments presented herein, which illustrates a distributed network computing environment 900 capable of implementing various aspects of the disclosed technology. Fig. 9 As shown in the figure, one or more server computers 900A can be interconnected with a number of client computing devices via a communication network 820 (which can be any one or combination of a wired or wireless LAN, WAN, intranet, extranet, peer-to-peer network, virtual private network, Internet, Bluetooth communication network, proprietary low voltage communication network or other communication networks), such as, but not limited to: tablet computer 900B, game console 900C, smart watch 900D, telephone such as smart phone 900E, personal computer 900F, and AR / VR device 900G.
[0085] In a network environment where the communication network 820 is the Internet, for example, the server computer 900A can be a dedicated server computer operable to process and transmit data to and from the client computing devices 900B-900G via any of a number of known protocols, such as the Hypertext Transfer Protocol ("HTTP") or the Simple Object Access Protocol ("SOAP"). In addition, the networked computing environment 900 can utilize various data security protocols, such as Secure Sockets Layer ("SSL") or Pretty Good Privacy ("PGP"). Each of the client computing devices 900B-900G can be equipped with an operating system operable to support one or more computing applications or terminal sessions, such as a web browser (in Fig. 9 ) or other graphical user interface (not shown in Fig. 9 ) or a mobile desktop environment (not shown in Fig. 9 ), to obtain access to server computer 900A.
[0086] The server computer 900A can be communicatively coupled to other computing environments (in Fig. 9) and receives data about the interaction / resource network of participating users. In an exemplary operation, a user (in Fig. 9 ) can interact with computing applications running on client computing devices 900B-900G to obtain desired data and / or execute other computing applications.
[0087] Data and / or computing applications may be stored on one or more servers 900A and transmitted to collaborating users via client computing devices 900B-900G over the exemplary communication network 820. Fig. 9 900A) can request access to specific data and applications hosted in whole or in part on the server computer 900A. Such data can be transmitted between the client computing devices 900B-900G and the server computer 900A for processing and storage.
[0088] The server computer 900A is capable of hosting computing applications, processes and applets for generating, authenticating, encrypting and transmitting data and applications, and can be used with other server computing environments (in Fig. 9 not shown), third-party service providers (in Fig. 9 (not shown), network-attached storage ("NAS"), and storage area network ("SAN") collaborate to enable application / data transactions.
[0089] It should be appreciated that for ease of discussion, the Figure 8 The computational architecture shown in Fig. 9 It should also be appreciated that the computing architecture and distributed computing network can include and utilize many more computing components, devices, software programs, networking devices, and other components not specifically described herein.
[0090] The disclosure set forth herein also covers the subject matter set forth in the following clauses:
[0091] Clause 1. A computer-implemented method performed by a data processing system, the computer-implemented method comprising: retrieving meeting-specific data associated with a meeting, the meeting-specific data comprising data associated with attendees of the meeting; generating a meeting-specific statistical language model based on the meeting-specific data; using the meeting-specific statistical language model to generate a transcription of audio generated during the meeting; and providing the transcription of the audio generated during the meeting to one or more computing devices for presentation in a user interface (UI).
[0092] Clause 2. The computer-implemented method of Clause 1, wherein the meeting-specific data comprises one or more of: a message associated with the attendee of the meeting, an attachment to the message, a name of the attendee, or a file associated with the attendee.
[0093] Clause 3. The computer-implemented method of Clause 1 or 2, further comprising storing the meeting-specific statistical language model during a duration of the meeting.
[0094] Clause 4. The computer-implemented method of any of Clauses 1-3, wherein the generating of the meeting-specific statistical language model occurs in response to receiving a pre-meeting signal, the pre-meeting signal comprising data identifying a start time for the meeting.
[0095] Clause 5. The computer-implemented method of any of Clauses 1-4, wherein the pre-meeting signal is generated a predetermined amount of time before the start time of the meeting.
[0096] Clause 6. The computer-implemented method of any of Clauses 1-5, wherein the predetermined amount of time is calculated by retrieving data identifying the start time for the meeting on a predetermined calendar.
[0097] Clause 7. A computer-implemented method as described in any of clauses 1-6, wherein the meeting-specific statistical language model defines a probability distribution of word occurrences for words in the meeting-specific data, and wherein using the meeting-specific statistical language model to generate the transcription of audio generated during the meeting includes using the meeting-specific statistical language model to determine probabilities of spoken words in the audio and generating the transcription based on the probabilities.
[0098] Clause 8. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to: retrieve meeting-specific data associated with a meeting, the meeting-specific data comprising data associated with attendees of the meeting; generate a meeting-specific statistical language model using the meeting-specific data; and generate a transcription of audio of the meeting using the meeting-specific statistical language model.
[0099] Clause 9. The computer-readable storage medium of clause 8, further having stored thereon additional computer-executable instructions for transmitting the transcription of the audio of the meeting to one or more computing devices for presentation in a user interface (UI) during the time of the meeting.
[0100] Clause 10. The computer-readable storage medium of clause 8 or 9, wherein the meeting-specific data comprises one or more of: a message associated with the attendee of the meeting, an attachment to the message, a name of the attendee, or a file associated with the attendee.
[0101] Clause 11. The computer-readable storage medium of any of Clauses 8-10, wherein the transcription of the audio of the conference is further generated at least in part using an underlying statistical language model.
[0102] Clause 12. The computer-readable storage medium of any of Clauses 8-11, wherein the generating of the meeting-specific statistical language model occurs in response to receiving a pre-meeting signal, the pre-meeting signal comprising data identifying a start time for the meeting.
[0103] Clause 13. The computer-readable storage medium of any of clauses 8-12, wherein the pre-meeting signal is generated a predetermined amount of time before the start time of the meeting.
[0104] Clause 14. The computer-readable storage medium of any of Clauses 8-13, wherein the pre-meeting signal is generated in response to receiving a meeting change event notification.
[0105] Clause 15. An apparatus for generating a transcription of audio of a conference, the apparatus comprising: means for retrieving conference-specific data associated with the conference, the conference-specific data comprising data associated with attendees of the conference; means for generating a conference-specific statistical language model based on the conference-specific data; and means for generating a transcription of the audio of the conference using the conference-specific statistical language model.
[0106] Clause 16. The apparatus of Clause 15, further comprising means for providing the transcription of the audio of the conference to one or more computing devices for presentation in a user interface (UI) during the conference.
[0107] Clause 17. The apparatus of clause 15 or 16, wherein the meeting specific data associated with the meeting is retrieved in response to receiving a signal generated on a predefined schedule prior to the meeting.
[0108] Clause 18. The apparatus of any of Clauses 15-17, wherein the meeting-specific data associated with the meeting is retrieved in response to receiving an event notification before or after the start of the meeting.
[0109] Clause 19. The apparatus of any of clauses 15-18, wherein the conference-specific data comprises one or more of: a message associated with the attendee of the conference, an attachment to the message, a name of the attendee, or a file associated with the attendee.
[0110] Clause 20. The apparatus of any of Clauses 15-19, wherein the transcription of the audio of the conference is further generated at least in part using an underlying statistical language model.
[0111] Although the techniques presented herein are described in language specific to structural features and / or methodological actions, it should be understood that the appended claims are not necessarily limited to the described features or actions. Instead, the features and actions are described as exemplary embodiments of such techniques. Furthermore, the above subject matter may be implemented as a computer-controlled device, a computer process, a computing system, or as an article of manufacture, such as a computer-readable storage medium.
[0112] The operations of the exemplary methods presented herein are illustrated in individual blocks and summarized with reference to these blocks. The methods are illustrated as a logical flow of blocks, each of which may be one or more operations that can be implemented in hardware, software, or a combination thereof. In the context of software, an operation represents a computer-executable instruction stored on one or more computer-readable media that, when executed by one or more processors, enables the one or more processors to perform the operation.
[0113] Typically, computer executable instructions include routines, programs, objects, modules, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations may be performed in any order, combined in any order, subdivided into multiple sub-operations, and / or performed in parallel to implement the process. The process may be performed by resources associated with one or more devices and / or one or more instances of hardware logic, such as one or more internal or external CPUs or GPUs, and hardware logic such as FPGAs, DSPs, or other types of accelerators.
[0114] All methods and processes described above can be embodied in software code modules executed by one or more general-purpose computers or processors and fully automated via them. The code modules can be stored in any type of computer-readable storage medium or other computer storage device. Some or all methods can alternatively be embodied in dedicated computer hardware.
[0115] Conditional language (e.g., particularly "can," "would," "might," or "may") should be understood in the context of introducing that certain examples include certain features, elements, and / or steps while other examples do not, unless otherwise specifically stated. Thus, such conditional language is generally not intended to imply that one or more examples require certain features, elements, and / or steps in any way, or that one or more examples necessarily include logic for determining (with or without user input or prompting) that certain features, elements, and / or steps are included in or will be performed by any particular example. Connective language (e.g., the phrase "at least one of X, Y, or Z") should be understood to introduce that an item, clause, etc. can be X, Y, or Z, or a combination thereof, unless otherwise specifically stated.
[0116] Any routine description, element or box in the flowchart described in this article and / or illustrated in the drawings should be understood to represent a module, paragraph or portion of code that includes one or more executable instructions for implementing a specific logical function or element in the routine. Alternative implementations are included within the scope of the examples described in this article, in which elements or functions can be deleted or executed out of the order shown or described, including substantially synchronously or in reverse order, according to functions that those skilled in the art will understand as involved.
[0117] It should be emphasized that many changes and modifications may be made to the above examples, the elements of which are to be understood as among other acceptable examples. All such modifications and variations are intended to be included herein, within the scope of the present disclosure and protected by the following claims.
Claims
1. A computer-implemented method performed by a data processing system, the computer-implemented method comprising: retrieving conference-specific data associated with a conference, the conference-specific data comprising data associated with attendees of the conference, wherein the conference-specific data is retrieved within a predetermined time period prior to a start time of the conference; generating a conference-specific statistical language model based on the conference-specific data, wherein the conference-specific statistical language model includes terms derived from an analysis of the conference-specific data; generating a transcription of audio generated during the meeting using the meeting-specific statistical language model; and The transcription of the audio generated during the conference is provided to one or more computing devices for presentation in a user interface (UI).
2. The computer-implemented method of claim 1 , wherein: The conference-specific data includes one or more of: a message associated with the attendee of the conference, an attachment to the message, a name of the attendee, or a file associated with the attendee. 3 . The computer-implemented method of claim 1 , further comprising storing the meeting-specific statistical language model during a duration of the meeting.
4. The computer-implemented method of claim 1 , wherein: The generating of the meeting-specific statistical language model occurs in response to receiving a pre-meeting signal including data identifying a start time for the meeting.
5. The computer-implemented method of claim 4, wherein: The pre-meeting signal is generated a predetermined amount of time before the start time of the meeting.
6. A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, cause the processor to: retrieving conference-specific data associated with a conference, the conference-specific data comprising data associated with attendees of the conference, wherein The meeting-specific data is retrieved within a predetermined time period prior to a start time of the meeting; generating a meeting-specific statistical language model using the meeting-specific data, wherein the meeting-specific statistical language model includes terms derived from an analysis of the meeting-specific data; and A transcription of the audio of the meeting is generated using the meeting-specific statistical language model.
7. A computer-readable storage medium according to claim 6, having additional computer-executable instructions stored thereon, the additional computer-executable instructions being used to transmit the transcription of the audio of the meeting to one or more computing devices for presentation in a user interface (UI) during the time of the meeting.
8. The computer-readable storage medium according to claim 6, wherein: The conference-specific data includes one or more of: a message associated with the attendee of the conference, an attachment to the message, a name of the attendee, or a file associated with the attendee.
9. The computer-readable storage medium according to claim 6, wherein: The transcription of the audio of the conference is further generated at least in part using an underlying statistical language model.
10. The computer-readable storage medium of claim 6, wherein: The generating of the meeting-specific statistical language model occurs in response to receiving a pre-meeting signal including data identifying a start time for the meeting.
11. The computer-readable storage medium according to claim 6, wherein: The pre-meeting signal is generated a predetermined amount of time before the start time of the meeting or in response to receiving a meeting change event notification.
12. An apparatus for generating a transcription of audio of a conference, the apparatus comprising: means for retrieving conference specific data associated with the conference, the conference specific data comprising data associated with attendees of the conference, wherein the conference specific data is retrieved within a predetermined time period prior to a start time of the conference; means for generating a meeting-specific statistical language model based on the meeting-specific data, wherein the meeting-specific statistical language model includes terms derived from an analysis of the meeting-specific data; and Module for generating a transcription of the audio of the meeting using the meeting-specific statistical language model.
13. The apparatus of claim 12, further comprising means for providing the transcription of the audio of the conference to one or more computing devices for presentation in a user interface (UI) during the conference.
14. The device according to claim 12, wherein: The meeting specific data associated with the meeting is retrieved in response to receiving a signal generated on a predefined schedule prior to the meeting.
15. The device according to claim 12, wherein: The transcription of the audio of the conference is further generated at least in part using an underlying statistical language model.
