Privacy-aware conference room transcription from audio-visual streams

Abstract

A method for privacy-aware transcription includes receiving an audiovisual signal including audio data and image data of a speech environment and a privacy request from a participant in the speech environment, wherein the privacy request indicates a privacy condition for the participant. The method further includes segmenting the audio data into a plurality of segments. For each segment, the method includes determining an identity of a speaker of a corresponding segment of the audio data based on the image data, and determining whether the identity of the speaker of the corresponding segment includes a participant associated with a privacy condition. When the identity of the speaker of the corresponding segment includes the participant, the method includes applying the privacy condition to the corresponding segment. The method also includes processing the plurality of segments of the audio data to determine a transcript of the audio data.

Description

Technical Field

[0001] The present disclosure relates to privacy-aware conference room transcription from audiovisual streams. Background Art

[0002] Speaker diarization is the process of dividing an input audio stream into homogeneous segments based on speaker identity. In environments with multiple speakers, speaker diarization answers the question "who spoke when?" and has various applications, including multimedia information retrieval, speaker turn analysis, and audio processing, to name a few. In particular, speaker diarization systems are able to produce speaker boundaries that have the potential to significantly improve acoustic speech recognition accuracy. Summary of the invention

[0003] One aspect of the present disclosure provides a method for generating a privacy-aware conference room transcript from a content stream. The method includes receiving an audiovisual signal including audio data and image data at data processing hardware. The audio data corresponds to voice utterances from a plurality of participants in a voice environment, and the image data represents faces of the plurality of participants in the voice environment. The method also includes receiving a privacy request from a participant of the plurality of participants at the data processing hardware. The privacy request indicates a privacy condition associated with the participant in the voice environment. The method further includes segmenting the audio data into a plurality of segments by the data processing hardware. For each segment of the audio data, the method includes determining, by the data processing hardware, an identity of a speaker of the corresponding segment of the audio data from the plurality of participants based on the image data. For each segment of the audio data, the method also includes determining, by the data processing hardware, whether the identity of the speaker of the corresponding segment includes a participant associated with the privacy condition indicated by the received privacy request. When the identity of the speaker of the corresponding segment includes a participant, the method includes applying the privacy condition to the corresponding segment. The method further includes processing, by the data processing hardware, the plurality of segments of the audio data to determine a transcript of the audio data.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, applying the privacy condition to the corresponding segment includes deleting the corresponding segment of the audio data after determining the transcript. Additionally or alternatively, applying the privacy condition to the corresponding segment may include enhancing the corresponding segment of the image data to visually hide the identity of the speaker of the corresponding segment of the audio data.

[0005] In some examples, for each portion of the transcript corresponding to one of the segments of the audio data to which the privacy condition is applied, processing the multiple segments of the audio data to determine the transcript of the audio data includes modifying the corresponding portion of the transcript to not include the identity of the speaker. Optionally, for each segment of the audio data to which the privacy condition is applied, processing the multiple segments of the audio data to determine the transcript of the audio data may include omitting the corresponding segment of the transcribing audio data. The privacy condition may include a content-specific condition indicating a type of content to be excluded from the transcript.

[0006] In some configurations, determining, from among the plurality of participants, an identity of a speaker of the corresponding segment of the audio data includes determining a plurality of candidate identities of the speaker based on the image data. Here, for each of the plurality of candidate identities, a confidence score is generated, the confidence score indicating a likelihood that a face of the corresponding candidate identity based on the image data includes a speaking face of the corresponding segment of the audio data. In this configuration, the method includes selecting the identity of the speaker of the corresponding segment of the audio data as the candidate identity of the plurality of candidate identities associated with the highest confidence score.

[0007] In some implementations, the data processing hardware resides on a device local to at least one of the plurality of participants. The image data may include high definition video processed by the data processing hardware. Processing the plurality of segments of audio data to determine a transcript of the audio data may include processing the image data to determine the transcript.

[0008] Another aspect of the present disclosure provides a system for privacy-aware transcription. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving an audiovisual signal including audio data and image data. The audio data corresponds to speech utterances from a plurality of participants in a speech environment, and the image data represents faces of a plurality of participants in the speech environment. The operations also include receiving a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the speech environment. The method further includes segmenting the audio data into a plurality of segments. For each segment of the audio data, the operation includes determining an identity of a speaker of a corresponding segment of the audio data from the plurality of participants based on the image data. For each segment of the audio data, the method also includes determining whether the identity of the speaker of the corresponding segment includes a participant associated with the privacy condition indicated by the received privacy request. When the identity of the speaker of the corresponding segment includes a participant, the operation includes applying the privacy condition to the corresponding segment. The operations further include processing the plurality of segments of the audio data to determine a transcript of the audio data.

[0009] This aspect can include one or more of the following optional features. In some examples, applying the privacy condition to the corresponding segment includes deleting the corresponding segment of the audio data after determining the transcript. Optionally, applying the privacy condition to the corresponding segment can include enhancing the corresponding segment of the image data to visually hide the identity of a speaker of the corresponding segment of the audio data.

[0010] In some configurations, processing the plurality of segments of audio data to determine a transcript of the audio data includes, for each portion of the transcript corresponding to one of the segments of the audio data to which the privacy condition is applied, modifying the corresponding portion of the transcript to not include an identity of the speaker. Additionally or alternatively, processing the plurality of segments of audio data to determine a transcript of the audio data may include, for each segment of the audio data to which the privacy condition is applied, omitting to transcribe the corresponding segment of the audio data. The privacy condition may include a content-specific condition indicating a type of content to be excluded from the transcript.

[0011] In some implementations, determining, from among the plurality of participants, an identity of a speaker of the corresponding segment of audio data includes determining a plurality of candidate identities of the speaker based on the image data. The implementation includes generating, for each of the plurality of candidate identities, a confidence score indicating a likelihood that a face of the corresponding candidate identity based on the image data includes a face of a speaker of the corresponding segment of audio data. The implementation also includes selecting the identity of the speaker of the corresponding segment of audio data as the candidate identity of the plurality of candidate identities associated with the highest confidence score.

[0012] In some examples, the data processing hardware resides on a device local to at least one of the plurality of participants. The image data may include high definition video processed by the data processing hardware. Processing the plurality of segments of the audio data to determine a transcript of the audio data may include processing the image data to determine the transcript.

[0013] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A is a schematic diagram of an exemplary gathering environment with a transcriber.

[0015] Figure 1B-1E is a privacy-aware transcriber Figure 1A Schematic diagram of an exemplary gathering environment.

[0016] Figure 2A and 2B is a schematic diagram of an exemplary transcriber.

[0017] Figure 3 is used in Figure 1A A flow chart of an exemplary arrangement of operations of a method of transcribing content within a gathering environment.

[0018] Figure 4 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0019] Figure 5 is a schematic diagram of an exemplary profile stored in memory hardware accessible to a transcriber.

[0020] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0021] The privacy of data used and generated by a video conferencing system is an important aspect of such a system. Conference participants may have their own personal views on the privacy of audio and video data obtained during a conference. Therefore, there is a technical problem of how to provide a video conferencing system that can accurately generate transcripts for video conferencing meetings while also meeting such privacy requirements in a reliable and accurate manner. The embodiments of the present disclosure provide a technical solution by enabling participants of the conference to set their own privacy configurations (e.g., opt-in or opt-out of various features of the video conferencing system), and then the video conferencing system accurately and effectively implements the expectations of the participants, because the video conferencing system identifies verbal contributions from participants based not only on the audio captured during the conference but also on the video captured during the conference when generating the transcript - this ensures a higher accuracy of identification of contributors to the video conference, thereby being able to improve the accuracy of the transcript while enabling the accurate and reliable implementation of the customized privacy requirements of the participants. In other words, a more accurate, reliable and flexible video conferencing system is provided.

[0022] Furthermore, in some implementations, the process of generating a transcript of a video conference is performed locally for one or more participants of the video conference, e.g., via a device in the same room as those participants. In other words, in such embodiments, the process of generating the transcript is not performed remotely, such as via one or more remote / cloud servers. This helps ensure that certain privacy expectations can be met, while also ensuring that locally captured full / original resolution and full / original quality video data is available for use in identifying speakers during the video conference (as opposed to a remote server operating on lower resolution and / or lower quality video, which may reduce the accuracy of speaker identification).

[0023] In a gathering environment (also commonly referred to as an environment), people gather together to communicate thoughts, ideas, schedules or other concerns. The gathering environment acts as a shared space for its participants. The shared space can be a physical space, such as a conference room or classroom, a virtual space (e.g., a virtual conference room), or any combination thereof. The environment can be a centralized location (e.g., locally hosted) or a decentralized location (e.g., virtually hosted). For example, an environment is a single room where participants gather, such as a conference room or classroom. In some embodiments, an environment is linked together to form more than one shared space for a gathering of participants. For example, a meeting has a host location (e.g., where the coordinator or speaker of the meeting may be located) and one or more remote locations that participate in the meeting (e.g., using a real-time communication application). In other words, an enterprise hosts a meeting from an office in Chicago, but other offices of the enterprise (e.g., in San Francisco or New York) participate in the meeting remotely. For example, there are many enterprises with large meetings across several offices, where each office has a meeting space to participate in the meeting. This is especially true because team members are distributed across the company (i.e., in more than one location) or even remote work is becoming more and more common. Furthermore, as applications become more robust for real-time communications, environments may be hosted for remote offices, remote employees, remote partners (eg, corporate partners), remote customers, etc. Thus, environments have evolved to accommodate a wide variety of convening organizational efforts.

[0024] Typically, as a space for communication, the environment hosts multiple participants. Here, each participant can contribute audio content (e.g., audible words spoken by speaking) and / or visual content (e.g., the actions of the participant) while present in the environment. In the case of more than one participant in the environment, it is beneficial to track and / or record the participation of any or all participants. This is especially true when the environment accommodates a wide range of gatherings and organizing work. For example, when the Chicago office hosts a meeting with both the New York office and the San Francisco office remotely present, it may be difficult for someone in the Chicago office to identify the speaker in one of the remote locations. To illustrate, the Chicago office may include a video feed that captures a conference room in each office away from the Chicago office. Even with the video feed, participants in the Chicago office may not be able to distinguish all participants in the New York office. For example, a speaker in the New York office is located far away from a camera associated with the video feed, making it difficult for participants in the Chicago office to identify who is the speaker in the New York office. This may also be difficult when Chicago-based participants are not familiar with other participants in the meeting (e.g., cannot identify the speaker by his / her voice). When the speaker cannot be identified, this may be problematic because the identity of the speaker may be a key component during the meeting. In other words, it may be important to identify the speaker (or source of content) to understand the main points / deliverables or generally understand who shared what content. For example, if Sally in the New York office takes on an action item that is deliverable to Johnny in the Chicago office, but Johnny cannot recognize that Sally took on the action item, Johnny may have difficulty following up on the action item later. In another scenario, because Johnny cannot recognize that Sally took on the action item, Johnny may incorrectly identify that Tracy (also in the New York office, for example) took on the action item. This is also true at a basic level of a simple conversation between participants. If Sally talks about a topic, but Johnny thinks it is Tracy who is talking, Johnny may cause confusion when he asks Tracy to join in on the topic later in the meeting.

[0025] Another problem may arise when the speaker discusses names, acronyms, and / or jargon that the other participant is unfamiliar with and / or does not fully understand. In other words, Johnny may be discussing a problem that occurred with a transportation vehicle used during a shipment. Pete may interject to help solve Johnny's problem by saying, "Oh, you will want to speak with Teddy in logistics about that." If Johnny is not familiar with Teddy and / or the organizational work team, Johnny may take notes about the conversation with Freddie instead of Teddy. This may also occur with acronyms or other jargon used in a given industry. For example, if the Chicago office is holding a meeting with a Seattle company, where the Chicago office hosts the meeting and the Seattle company participates in the meeting remotely, participants in the Chicago office may use acronyms and / or jargon that the Seattle company is unfamiliar with. Unfortunately, without a record or transcription of the content presented by the Chicago office, the Seattle company may not understand the meeting (e.g., resulting in a poor meeting). Additionally or alternatively, poor connectivity between locations or with the conference hosting platform may also complicate things for participants as they attempt to understand content during the conference.

[0026] To overcome these problems, there is a transcription device in the environment, which (e.g., in real time) generates a transcript of the content that occurs in the environment. When generating a transcript, the device can identify the speaker (i.e., the participant who generates the audio content) and / or associate the content with the participant who also exists in the environment. Utilizing the transcript of the content presented in the environment, the transcription device can remember the main points and / or deliverables, and provide a record of who initiated the content accessible to the participant for reference. For example, the participant can refer to the transcript during the meeting (e.g., in real time or substantially in real time) or at a later time after the meeting. In other words, Johnny can refer to the display of the transcript generated by the transcription device to identify that Teddy (rather than Freddie) is the person he needs to talk to in the organization's work, and he should follow up with Sally (rather than Tracy) about the action item.

[0027] Unfortunately, although the transcript may solve some problems encountered in the environment, it raises questions about privacy. Here, privacy refers to having a state of being free from observation on the transcript generated by the transcription device. Although there may be many different types of privacy, some examples include content privacy or identity privacy. Here, content privacy is based on content, so that it is expected that some sensitive content is not remembered in a written or human-readable format (e.g., confidential content). For example, a part of a meeting may include audio content about another employee who does not exist in the meeting (e.g., the manager discusses the human resources issues that arise). In this example, the participants in the meeting will prefer not to transcribe or otherwise remember this part of the meeting about this other employee. This can also include not remembering the audio content including the content about this other employee. Here, since traditional transcription equipment transcribes content indiscriminately, the meeting will not be able to utilize traditional transcription equipment at least during this part of the meeting.

[0028] Identity privacy refers to the privacy that seeks to maintain the anonymity of the content source. For example, the transcript usually includes a mark identifying the source of the transcribed content within the transcript. For example, the speaker who marks the transcribed content can be called speaker log-in to answer "who said what" and "who spoke when". When the identity of the content source is sensitive or the source (e.g., participant) who generates the content prefers to mask his / her identity for any reason (e.g., personal reasons), the source does not want the label to be associated with the transcribed content. Note that here, unlike content privacy, the source does not mind the content being revealed in the transcript, but does not want the identifier (e.g., label) to associate the content with the source. Since traditional transcription equipment lacks the ability to adapt to these privacy issues, even if the above benefits are abandoned, participants can choose not to use transcription equipment. In order to maintain these benefits and / or protect the privacy of participants, the environment can include a privacy-aware transcription device called a transcriber. In another example, when a camera is capturing a video of a speaker who wants to remain anonymous, the speaker can choose not to have their recorded image (e.g., face) remembered. This may include distorting the video / image frame of the speaker's face and / or overlaying a graphic that masks the speaker's identity so that other individuals in the conference cannot visually identify the speaker. Additionally or alternatively, the audio of the speaker's voice may be distorted (e.g., by passing the audio through a vocoder) to mask the speaker's voice in a manner that anonymizes the speaker.

[0029] In some embodiments, privacy concerns are further enhanced by processing privacy on the device during transcription so that the transcript does not leave the confines of the gathering environment (e.g., a conference room or classroom) that provides a shared space for its participants. In other words, by using the transcriber to generate the transcript on the device, speaker tags that identify speakers who want to remain anonymous can be removed on the device to alleviate any concerns that the identities of those speakers will be exposed / compromised if processing of the transcript occurs on a remote system (e.g., a cloud environment). In other words, there is no unedited transcript generated by the transcriber that could be shared or stored to compromise the privacy of the participants.

[0030] Another technical effect of performing audio-video transcription (e.g., Audio-Video Automatic Voice Recognition (AVASR)) on the device is reduced bandwidth requirements, as the audio and image data (also referred to as video data) can be retained locally on the device without the need to transmit it to a remote cloud server. For example, if the video data is to be transmitted to the cloud, it may first need to be compressed for transmission. Therefore, another technical effect of performing video matching on the user device itself is that the video data matching can be performed using uncompressed (highest quality) video data. The use of uncompressed video data makes it easier to identify a match between the audio data and the speaker's face, so that speaker labels assigned to transcribed portions of the audio data spoken by speakers who do not want to be identified can be anonymized. At the same time, video data capturing the faces of individuals who do not want to be identified can be enhanced / distorted / blurred to mask these individuals so that if the video recording is shared, they cannot be visually identified. Similarly, audio data representing words spoken by these individuals may be distorted to anonymize the spoken speech of these individuals who are not wanted to be identifiable. Reference Figures 1A-1E, environment 100 includes multiple participants 10, 10a-j. Here, environment 100 is a host conference room, where six participants 10a-f participate in a meeting (e.g., a video conference) in the host conference room. Environment 100 includes a display device 110, which receives a content feed 112 (also known as a multimedia feed, content stream, or feed) from a remote system 130 via a network 120. The content feed 112 can be an audio feed 218 (i.e., audio data 218 such as audio content, audio signal, or audio stream), a visual feed 217 (i.e., image data 217, such as video content, video signal, or video stream), or some combination of the two (e.g., also known as an audio-visual feed, audio-visual signal, or audio-visual stream). Display device 110 includes a display 111 capable of displaying video content 217 and a speaker for audible output of audio content 218, or communicates with it. Some examples of display device 110 include computers, laptops, mobile computing devices, televisions, monitors, smart devices (e.g., smart speakers, smart displays, smart appliances), wearable devices, etc. In some examples, display device 110 includes audio-visual feeds 112 of other conference rooms participating in the meeting. For example, Figures 1A-1E Two feeds 112, 112a-b are depicted, where each feed 112 corresponds to a different remote conference room. Here, the first feed 112a includes three participants 10, 10g-i, while the second feed 112b includes a single participant 10, 10j (e.g., an employee working away from a home office). To continue with the previous example, the first feed 112a may correspond to a feed 112 from the New York office, the second feed 112b corresponds to a feed 112 from the San Francisco office, and the host conference room 100 corresponds to the Chicago office.

[0031] Remote system 130 can be a distributed system (e.g., a cloud computing environment or storage abstraction) with scalable / elastic resources 132. Resources 132 include computing resources 134 (e.g., data processing hardware) and / or storage resources 136 (e.g., memory hardware). In some embodiments, remote system 130 hosts (e.g., on computing resources 132) software for coordination environment 100. For example, computing resources 132 of remote system 130 execute software, such as a real-time communication application or a professional conferencing platform.

[0032] Continue to refer Figures 1A-1E, the environment 100 also includes a transcriber 200. The transcriber 200 is configured to generate a transcript 202 of content occurring within the environment 100. The content can come from a location where the transcriber 200 is located (e.g., a participant 10 in a conference room 100 having the transcriber 200) and / or from a content feed 112 that transmits the content to the transcriber 200. In some examples, the display device 110 transmits one or more content feeds 112 to the transcriber 200. For example, the display device 110 includes a speaker that outputs audio content 218 of the content feed 112 to the transcriber 200. In some implementations, the transcriber 200 is configured to receive the same content feed 112 as the display device 110. In other words, the display device 110 can act as an extension of the transcriber 200 by receiving the audio and video feeds of the content feed 112. For example, the display device 110 may include hardware 210, such as data processing hardware 212 and memory hardware 214 in communication with the data processing hardware 212, which enables the data processing hardware 212 to execute the transcriber 200. In this relationship, the transcriber 200 may receive the content feed 112 (e.g., audio and visual content / signals 218, 217) through a network connection, rather than only audibly capturing the audio content / signals 218 relayed through a peripheral device (such as a speaker) of the display device 110. In some examples, such a connection between the transcriber 200 and the display device 110 enables the transcriber 200 to seamlessly display the transcript 202 on a display / screen 111 of the display device 110 locally within the environment 100 (e.g., a hosting conference room). In other configurations, the transcriber 200 is located in the same local environment 110 as the display device 110, but corresponds to a computing device separate from the display device 110. In these configurations, the transcriber 200 communicates with the display device 110 via a wired or wireless connection. For example, the transcriber 200 has one or more ports that allow for wired / wireless connections, such that the display device 110 is used as a peripheral device for the transcriber 200. Additionally or alternatively, the application forming the environment 100 may be compatible with the transcriber 200. For example, the transcriber 200 is configured as an input / output (I / O) device within the application, such that audio and / or visual signals coordinated by the application are dialed to the transcriber 200 (e.g., in addition to the display device 110).

[0033] In some examples, the transcriber 200 (and optionally the display device 110) is portable, such that the transcriber 200 can be transferred between conference rooms. In some implementations, the transcriber 200 is configured with processing capabilities (e.g., processing hardware / software) to process audio and video content 112 and generate a transcript 202 when the content 112 is presented in the environment 100. In other words, the transcriber 200 is configured to process the content 112 (e.g., audio and / or visual content 218, 217) locally at the transcriber 200 to generate the transcript 202 without any additional remote processing (e.g., at the remote system 130). This type of processing is referred to herein as on-device processing. Unlike remote processing, which often uses low-fidelity compressed video on server-based applications due to bandwidth constraints, on-device processing may not be constrained by bandwidth, allowing the transcriber 200 to utilize more accurate high-definition video with high fidelity when processing visual content. Additionally, the on-device processing may allow for real-time tracking of the identity of the speaker without delays due to latency that would occur if the audio and / or visual signals 218, 217 were processed remotely to some extent (e.g., in a remote computing system 130 connected to the transcriber 200). To process content at the transcriber 200, the transcriber 200 includes hardware 210, such as data processing hardware 212 and memory hardware 214 in communication with the data processing hardware 212. Some examples of the data processing hardware 212 include a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

[0034] In some implementations, the transcriber 200 is executed on the remote system 130 by receiving the content 112 (audio and video data 217, 218) from each of the first and second feeds 112a-b and the feed 112 from the conference room environment 100. For example, the data processing hardware 134 of the remote system 130 may execute instructions stored on the memory hardware 136 of the remote system 130 for executing the transcriber 200. Here, the transcriber 200 may process the audio data 218 and the image data 217 to generate the transcript 202. For example, the transcriber 200 may generate the transcript 202 and transmit the transcript 202 to the display device 110 via the network 120 for display thereon. The transcriber 200 may similarly transmit the transcript 202 to a computing device / display device associated with the participants 10g-i corresponding to the first feed and / or the participant 10j corresponding to the second feed 10j.

[0035] In addition to the processing hardware 210, the transcriber 200 also includes peripheral devices 216. For example, to process audio content, the transcriber 200 includes an audio capture device 216, 216a (e.g., a microphone) that captures sounds (e.g., voice utterances) about the transcriber 200 and converts the sounds into audio signals 218 ( Figure 2A and 2B ) (or audio data 218 ). The transcriber 200 may then use the audio signal 218 to generate the transcript 202 .

[0036] In some examples, the transcriber 200 also includes an image capture device 216, 216b as a peripheral device 216. Here, the image capture device 216b (e.g., one or more cameras) can capture image data 217 ( Figure 2A and Figure 2B ) as an additional input source (e.g., visual input) that, in combination with the audio signal 218, helps identify which participant 10 in the multi-participant environment 100 is speaking (i.e., the speaker). In other words, by including both the audio capture device 216a and the image capture device 216b, the transcriber 200 can increase the accuracy of its speaker identification because the transcriber 200 can process the image data 217 captured by the image capture device 216b to identify visual features (e.g., facial features) that indicate which participant 10 of the multiple participants 10a-10j is speaking (i.e., generating the utterance 12) in a particular instance. In some configurations, the image capture device 216b is configured to capture 360 ​​degrees around the transcriber 200 to capture a panoramic view of the environment 100. For example, the image capture device 216b includes a camera array configured to capture a 360-degree view.

[0037] Additionally or alternatively, the use of image data 217 may improve transcript 202 when participant 10 has a speech impediment. For example, transcriber 200 may have difficulty generating a transcript for a speaker with a speech impediment that causes the speaker to have problems articulating speech. To overcome inaccuracies in transcript 202 caused by such articulation problems, it may be possible to (e.g., Figure 2A and Figure 2BThe transcriber 200 at the automatic speech recognition (ASR) module 230 of the embodiment of the present invention is aware of the articulation problem during the generation of the transcript 202. By being aware of the problem, the transcriber 200 can adapt to the problem by utilizing the image data 217 representing the face of the participant 10 while speaking to generate an improved or otherwise more accurate transcript 202, rather than the transcript 202 being based solely on the audio data 218 of the participant 10. Here, certain speech disorders may be evident in the image data 217 from the image capture device 216b. For example, in the case of speech dysarthria, a neuromuscular disorder causing lip movements that affect articulation may be identified in the image 217. In addition, techniques may be employed in which the image data 217 may be analyzed to correlate the lip movements of participants 10 having specific speech disorders with the intended speech of these participants 10, thereby improving automatic speech recognition in a manner that is not possible using the audio data 218 alone. In some implementations, by using the image 217 as an input to the transcriber 200 , the transcriber 200 identifies potential articulation issues and takes the issues into account to improve the generation of the transcript 202 during ASR.

[0038] In some embodiments, such as Figure 1B-1E , the transcriber 200 is privacy-aware such that a participant 10 may choose not to share any of his or her voice and / or image information (e.g., in the transcript 202 or visual feeds 112, 217). Here, one or more participants 10 transmit a privacy request 14 that indicates privacy conditions for the participant 10 during participation in the video conference environment 100. In some examples, the privacy request 14 corresponds to a configuration setting of the transcriber 200. The privacy request 14 may occur before, during, or at the beginning of a meeting or communication session with the transcriber 200. In some configurations, the transcriber 200 includes one or more privacy requests 14 (e.g., Figure 5 of the personal profiles 510, 510a-n) (e.g., Figure 514. The profile 500 shown in FIG. 14 is a profile 500 for a participant 10. Here, the profile 500 may be stored on the device (e.g., in the memory hardware 214) or stored off the device (e.g., in the remote storage resource 136) and accessed by the transcriber 200. The profile 500 may be configured prior to the communication session and may include an image of the face of the corresponding participant 10 (e.g., the image data 217) so that the participant 10 may be associated with the corresponding portion of the received video content 217. That is, when the video content 217 of the participant 10 in the content feed 112 matches the facial image associated with the personal profile 510, the personal profile 510 of the corresponding participant 10 may be accessed. Using the personal profile 510, the privacy settings of the participant may be applied during each communication session in which the participant 10 participates. In these examples, the transcriber 200 may recognize the participant 10 (e.g., based on the image data 217 received at the transcriber 200) and apply the appropriate settings for the participant 10. For example, the profile 500 may include an individual profile 510, 510b for a particular participant 10, 10b indicating that the particular participant 10b does not mind being seen (i.e., included in the visual feed 217), but does not want to be heard (i.e., not included in the audio feed 218) or have his / her words 12 transcribed (i.e., not included in the speech in the transcript 202), while another individual profile 510, 510c for another participant 10, 10c may not want to be seen (i.e., not included in the visual feed 217), but may have his / her words recorded and / or transcribed (i.e., included in the audio feed 218 and included in the transcript 202).

[0039] refer to Figure 1B , the third participant 10c has submitted a privacy request 14 with a privacy condition (i.e., a privacy request 14 for identity privacy) indicating that the third participant 10c does not mind being seen or heard, but does not want the transcript 202 to include an identifier 204 for the third participant 10c (e.g., a label of the speaker's identity) when the third participant 10c is speaking. In other words, the third participant 10c does not want his or her identity to be shared or stored; therefore, the third participant 10c selects that the transcript 202 does not include an identifier 204 associated with the third participant 10c that reveals his or her identity. Here, although Figure 1B Transcript 202 is illustrated with an edited grey box in which identifier 204 for speaker 3 is present, but transcriber 200 may also remove identifier 204 entirely or obscure identifier 204 in other ways that prevent the identity of the speaker associated with privacy request 14 from being revealed by transcriber 200. In other words, Figure 1B Transcriber 200 is illustrated as modifying a portion of transcript 202 to not include the identity of the speaker (eg, by removing or obfuscating identifier 204).

[0040] Figure 1C Similar to Figure 1B , except that the third participant 10c transmitting the privacy request 14 requests not to be seen in any visual feed 112, 217 of the environment 100 (e.g., another form of identity privacy). Here, the requesting participant 10c may not mind being heard, but prefers to visually hide his or her visual identity (i.e., not share or store his or her visual identity in the visual feed 112, 217). In this case, the transcriber 200 is configured to blur, distort, or otherwise obscure the visual presence of the requesting participant 10c throughout the communication session between the participants 10, 10a-10j. For example, in any instance, the transcriber 200 determines the location of the requester 10c based on the image data 217 received from the one or more content feeds 112, and applies the abstraction 119 to any physical features of the requester transmitted by the transcriber 200 (e.g., blur). That is, when the image data 217 is displayed on the screen 111 of the display device 110 and on screens in the remote environment associated with the participants 10g-10j, the abstraction 119 is overlaid on at least the face of the requester 10c such that the requester 10c cannot be visually identified. In some examples, the personal profile 510 of the participant 10 identifies whether the participant 10 wants to be blurred or obscured (i.e., distorted) or removed entirely (e.g., as in Figure 5 ). Thus, transcriber 200 is configured to enhance, modify, or remove portions of video data 217 to conceal the visual identities of the participants.

[0041] on the contrary, Figure 1DAn example is illustrated in which a privacy request 14 from a third participant 10c requests that the transcriber 200 not track a visual representation of the third participant 10c or voice information of the third participant 10c. As used herein, "voice information" refers to audio data 218 corresponding to the utterance 12 spoken by the participant 10c and a transcript 202 identified from the audio data 218 corresponding to the utterance 12 spoken by the participant 10c. In this example, the participant 10c can be heard during the meeting, but the transcriber 200 does not remember the participant 10c audibly or visually (e.g., through the video feed 217 or in the transcript 202). The method can protect the privacy of the participant 10c by not having a record of any voice information of the participant 10c in the transcript 202 or any identifier 204 identifying the participant 10c in the transcript 202. For example, the transcriber 200 may completely omit the portions of text in the transcript 202 that transcribe the utterance 12 spoken by the participant 10c, or the transcriber 202 may leave these portions of text but not apply the identifier 204 that identifies the participant 10c. However, the transcriber 200 may apply some other arbitrary identifier that does not personally identify the participant 10c but instead merely demarcates these portions of text in the transcript 202 from other portions that correspond to the utterance 12 spoken by the other participants 10a, 10b, 10d-10j. In other words, the participant 10 may request (e.g., via a privacy request 14) that the transcript 202 and any other records generated by the transcriber 200 have no record of the participant's participation in the communication session.

[0042] Compared to identity privacy request 14, Figure 1EA content privacy request 14 is depicted. In this example, a third participant 10c transmits a privacy request 14 that the transcriber 200 not include any content from the third participant 10c in the transcript 202. Here, the third participant 10c makes such a privacy request 14 because the third participant 10c will be discussing sensitive content (e.g., confidential information) during the meeting. Due to the sensitive nature of the content, the third participant 10c takes the following precautions: the transcriber 200 does not remember the audio content 218 associated with the third participant 10c in the transcript 202. In some embodiments, the transcriber 200 is configured to receive a privacy request 14 that identifies (e.g., by keywords) the type of content that one or more participants 10 do not want to include in the transcript 202, and determines when that type of content occurs during the communication session so as to exclude it from the transcript 202. In these embodiments, not all audio content 218 from a particular participant 10 is excluded from the transcript 202, only content-specific audio is excluded so that the particular participant can still discuss other types of content and be included in the transcript 202. For example, the third participant 10c transmits a privacy request 14 requesting that the transcriber 200 not transcribe audio content about Mike. In this case, when the third participant 10c discusses Mike, the transcriber 200 does not transcribe the audio content 218, but when the third participant talks about other topics (e.g., the weather), the transcriber 200 does transcribe the audio content 218. The participant 10c can similarly set a time boundary so that the transcriber 200 does not remember any audio content 218 for a period of time (e.g., the next 2 minutes).

[0043] Figure 2A and 2Bis an example of a transcriber 200. The transcriber 200 generally includes a logification module 220 and an ASR module 230 (e.g., an AVASR module). The logification module 220 is configured to receive audio data 218 corresponding to an utterance 12 from a participant 10 of a communication session (e.g., captured by an audio capture device 216a) and image data 217 representing a face of the participant 10 of the communication session, segment the audio data 218 into a plurality of segments 222, 222a-n (e.g., fixed-length segments or variable-length segments), and generate a logification result 224, which includes a corresponding speaker label 226 assigned to each segment 222 based on the audio data 218 and the image data 217 using a probabilistic model (a probabilistic generative model). In other words, the logification module 220 includes a series of speaker identification tasks with short utterances (e.g., segments 222), and determines whether two segments 222 of a given conversation are spoken by the same participant 10. At the same time, the logging module 220 can perform a facial tracking routine to identify which participant 10 is speaking during which segment 222 to further optimize speaker identification. The logging module 220 is then configured to repeat the process for all segments 222 of the conversation. Here, the logging results 224 provide the received audio data 218 with time-stamped speaker tags 226, 226a-e, which not only identify who is speaking during a given segment 222, but also identify when a speaker change occurs between adjacent segments 222. Here, the speaker tags 226 can be used as identifiers 204 within the transcript 202.

[0044] In some examples, the transcriber 200 receives the privacy request 14 at the logging module 220. Because the logging module 220 identifies the speaker tag 226 or the identifier 204, the logging module 220 can advantageously resolve the privacy request 14 corresponding to the identity-based privacy request 14. In other words, when the participant 10 is the speaker, the logging module 220 receives the privacy request 14 when the privacy request 14 requests that the participant 10 not be identified by the identifier 204 such as the tag 226. When the logging module 220 receives the privacy request 14, the logging module 220 is configured to determine whether the participant 10 corresponding to the request 14 matches the tag 226 generated for the given segment 222. In some examples, an image of the face of the participant 10 can be used to associate the participant 10 with the tag 226 for the participant 10. When the tag 226 for the segment 222 matches the identity of the participant 10 corresponding to the request 14, the logging module 220 can prevent the transcriber 200 from applying the tag 226 or the identifier 204 to the corresponding portion of the resulting transcript 202 that transcribes the particular segment 222 into text. When the tag 226 for the segment 222 fails to match the identity of the participant 10 corresponding to the request 14, the logging module 220 can allow the transcriber to apply the tag 226 and the identifier 204 to the portion of the resulting transcript 202 that transcribes the particular segment into text. In some embodiments, when the logging module 220 receives the request 14, the ASR module 230 is configured to wait for the audio data 218 from the utterance 12 to be transcribed. In other embodiments, the ASR module 230 transcribes in real time, and the resulting transcript 202 removes the tag 226 and the identifier 204 in real time for any participant 10 who provided a privacy request 14 that opted not to have their voice information transcribed. Optionally, the logging module 220 may further distort the audio data 218 associated with those participants 10 seeking privacy such that their spoken voice is altered in a manner that cannot be used to identify the participant 10 .

[0045] The ASR module 230 is configured to receive audio data 218 corresponding to the utterance 12 and image data 217 representing the face of the participant 10 when the utterance 12 is being spoken. Using the image data 217, the ASR module 230 transcribes the audio data 218 into corresponding ASR results 232. Here, the ASR results 232 refer to text transcriptions (e.g., transcripts 202) of the audio data 218. In some examples, the ASR module 230 communicates with the logging module 220 to utilize the logging results 224 associated with the audio data 218 to improve speech recognition based on the utterance 12. For example, the ASR module 230 can apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the logging results 224. Additionally or alternatively, the ASR module 230 and / or the logging module 220 (or some other component of the transcriber 200) can use the predicted timestamped speaker tags 226 for each segment 222 obtained from the logging results 224 to index the transcription 232 of the audio data 218. In other words, the ASR module 230 uses the speaker tags 226 from the logging module 220 to generate the identifiers 204 of the speakers within the transcript 202. Figures 1A-1E As shown, transcripts 202 of communication sessions within environment 100 may be indexed by speakers / participants 10 to associate portions of transcripts 202 with corresponding speakers / participants 10 in order to identify what each speaker / participant 10 said.

[0046] In some configurations, the ASR module 230 receives a privacy request 14 to the transcriber 200. For example, the ASR module 230 receives a privacy request 14 to the transcriber 200 whenever the privacy request 14 corresponds to a request 14 not to transcribe the speech of a particular participant 10. In other words, the ASR module 230 may receive a privacy request 14 whenever the request 14 is not a privacy request 14 based on a tag / identifier. In some examples, when the ASR module 230 receives the privacy request 14, the ASR module 230 first identifies the participant 10 corresponding to the privacy request 14 based on the speaker tag 226 determined by the logging module 220. Then, when the ASR module 230 encounters speech to be transcribed for the participant 10, the ASR module 230 applies the privacy request 14. For example, when the privacy request 14 requests not to transcribe the speech of the particular participant 10, the ASR module 230 does not transcribe any speech of the participant and waits for the speech of a different participant 10 to appear.

[0047] refer to Figure 2BIn some embodiments, the transcriber 200 includes a detector 240 for performing a facial tracking routine. In these embodiments, the transcriber 200 first processes the audio data 218 to generate one or more candidate identities for the speaker. For example, for each segment 222, the log module 220 may include a plurality of tags 226, 226a 1-3 In other words, the model can output multiple labels 226, 226a for each segment 222. 1-3 A probability model, wherein multiple labels 226, 226a 1-3 Each label 226 in is a potential candidate for identifying a speaker. Here, the detector 240 of the transcriber 200 uses the images 217, 217a-n captured by the image capture device 216b to determine which candidate identity has the best visual features indicating that he or she is the speaker of the particular segment 22. In some configurations, the detector 240 generates a score 242 for each candidate identity, wherein the score 242 indicates a confidence level that the candidate identity is the speaker based on an association between an audio signal (e.g., audio data 218) and a visual signal (e.g., captured images 217a-n). Here, the highest score 242 may indicate the greatest likelihood that the candidate identity is the speaker. Figure 2B In the example, the logging module 220 generates three tags 226a at a specific segment 222. 1-3 The detector 240 generates a score 242 (eg, shown as three scores 242 ) for each of these labels 226 based on the image 217 from the time when the segment 222 occurs in the audio data 218 . 1-3 ). Here, Figure 2B The highest score 242 is indicated by a bold square around the third label 226a3 associated with the third score 2423. When the transcriber 200 includes the detector 240, the best candidate identity may be communicated to the ASR module 230 to form the identifier 204 of the transcript 202.

[0048] Additionally or alternatively, the process may be reversed such that the transcriber 200 first processes the image data 217 to generate one or more candidate identities of the speaker based on the image data 217. Then, for each candidate identity, the detector 240 generates a confidence score 242 that indicates the likelihood that the face of the corresponding candidate identity includes a speaking face for the corresponding segment 222 of the audio data 218. For example, the confidence score 242 for each candidate identity indicates the likelihood that the face of the corresponding candidate identity includes a speaking face during the image data 217 at the time instance of the segment 222 corresponding to the audio data 218. In other words, for each segment 222, the detector 240 may score 242 whether the image data 217 corresponding to the participant 10 has a facial expression that is similar to or matches the facial expression of the speaking face. Here, the detector 240 selects the identity of the speaker of the corresponding segment of the audio data 218 with the highest confidence score 242 as the candidate identity.

[0049] In some examples, the detector 240 is part of the ASR module 230. Here, the ASR module 230 performs a face tracking routine by implementing an encoder front end having an attention layer configured to receive multiple video tracks 217a-n of the image data 217, whereby each video track is associated with a face of a corresponding participant. In these examples, the attention layer at the ASR module 230 is configured to determine a confidence score indicating a likelihood that the face of the corresponding person associated with the video face track includes a speaking face of the audio track. Additional concepts and features related to an audio-visual ASR module including an encoder front end with an attention layer for multi-speaker ASR recognition can be found in U.S. Provisional Patent Application 62 / 923,096, filed on October 18, 2019, the entire contents of which are incorporated herein by reference.

[0050] In some configurations, the transcriber 200 (e.g., at the ASR module 230) is configured to support a multi-language environment 100. For example, when the transcriber 200 generates the transcript 202, the transcriber 200 can generate the transcript 202 in different languages. This feature can enable the environment 100 to include a remote location that has one or more participants 10 that speak a different language than the host location. In addition, in some cases, the speaker in the meeting can be a non-native speaker or a speaker whose language of the meeting is not the speaker's first language. Here, the transcript 202 of the content from the speaker can help other participants 10 in the meeting understand the presented content. Additionally or alternatively, the transcriber 200 can be used to provide feedback to the speaker about his or her pronunciation. Here, by combining video and / or audio data, the transcriber 200 can indicate incorrect pronunciation (e.g., allowing the speaker to learn and / or adapt with the help of the transcriber 200). In this way, the transcriber 200 can provide a notification to the speaker that provides feedback about his / her pronunciation.

[0051] Figure 3is an exemplary arrangement of operations of a method 300 of transcribing content (e.g., at the data processing hardware 212 of the transcriber 200). At operation 302, the method 300 includes receiving an audiovisual signal 217, 218 including audio data 218 and image data 217. The audio data 218 corresponds to a voice utterance 12 from a plurality of participants 10, 10a-n in the speech environment 100, and the image data 217 represents faces of the plurality of participants 10 in the speech environment 100. At operation 304, the method 300 includes receiving a privacy request 14 from a participant 10 of the plurality of participants 10a-n. The privacy request 14 indicates a privacy condition associated with the participant 10 in the speech environment 100. At operation 306, the method 300 segments the audio data 218 into a plurality of segments 222, 222a-n. At operation 308, the method 300 includes performing operations 308, 308a-c on each segment 222 of the audio data 218. At operation 308a, for each segment 222 of the audio data 218, the method 300 includes determining, based on the image data 217, an identity of a speaker of the corresponding segment 222 of the audio data 218 from the plurality of participants 10a-n. At operation 308b, for each segment 222 of the audio data 218, the method 300 includes determining whether the identity of the speaker of the corresponding segment 222 includes the participant 10 associated with the privacy condition indicated by the received privacy request 14. At operation 308c, for each segment 222 of the audio data 218, when the identity of the speaker of the corresponding segment 222 includes the participant 10, the method 300 includes applying the privacy condition to the corresponding segment 222. At operation 310, the method 300 includes processing the plurality of segments 222a-n of the audio data 218 to determine a transcript 202 of the audio data 218.

[0052] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only when explicit authorization to do so is received from the relevant user.

[0053] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information associated with that user to provide permission or authorization as to whether information is to be collected and as to which portions of the information are to be collected. For example, one or more such control options may be provided to a user via a communications network. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined.

[0054] Figure 4 4 is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems and methods described in this document. Computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0055] The computing device 400 includes a processor 410 (e.g., data processing hardware), a memory 420 (e.g., memory hardware), a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 410 is capable of processing instructions for execution within the computing device 400, including instructions stored in the memory 420 or stored on the storage device 430, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 480 coupled to the high-speed interface 440. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used as appropriate. Furthermore, multiple computing devices 400 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a blade server bank, or a multi-processor system).

[0056] Memory 420 stores information non-temporarily within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0057] Storage device 430 can provide mass storage for computing device 400. In some embodiments, storage device 430 is a computer readable medium. In various embodiments, storage device 430 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid state storage device, or a device array, including a device in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied as an information carrier. The computer program product includes instructions for executing one or more methods, such as those described above, when executed. The information carrier is a computer or machine readable medium, such as a memory 420, a storage device 430, or a memory on a processor 410.

[0058] The high-speed controller 440 manages bandwidth-intensive operations of the computing device 400, while the low-speed controller 460 manages less bandwidth-intensive operations. This division of responsibilities is illustrative only. In some embodiments, the high-speed controller 440 is coupled to a memory 420, a display 480 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 450 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 460 is coupled to a storage device 430 and a low-speed expansion port 470. The low-speed expansion port 470, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.

[0059] As shown, computing device 400 may be implemented in a variety of different forms. For example, computing device 400 may be implemented as a standard server 400a or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0060] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to these devices.

[0061] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0062] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by dedicated logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices such as disks, magneto-optical disks, or optical disks for storing data, or be operably coupled to a large-capacity storage device to receive data from it or transmit data to it, or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, such as EPROMs, EEPROMs, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0063] To provide interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, to display information to the user, and an optional keyboard and pointing device, such as a mouse and trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0064] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A method for privacy-aware transcription, comprising: receiving, at data processing hardware, an audiovisual signal comprising audio data corresponding to voice utterances from a plurality of participants in a voice environment and image data representing faces of the plurality of participants in the voice environment; receiving, at the data processing hardware, a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the voice environment; dividing the audio data into a plurality of segments by the data processing hardware; For each segment of the audio data: determining, by the data processing hardware, an identity of a speaker of a corresponding segment of the audio data from among the plurality of participants based on the image data; determining, by the data processing hardware, whether the identities of the speakers of the corresponding segment include the participants associated with the privacy condition indicated by the received privacy request; as well as When the identity of the speaker of the corresponding segment includes the participant, applying the privacy condition indicated by the received privacy request to the corresponding segment, wherein applying the privacy condition indicated by the received privacy request to the corresponding segment includes omitting transcribing the corresponding segment based on the privacy condition; and processing the multiple segments of the audio data by the data processing hardware to determine a transcript of the audio data, wherein the transcript includes text corresponding to each other segment of the multiple segments except the corresponding segment and omitting text corresponding to the corresponding segment.

2. The method according to claim 1, wherein: Applying the privacy condition to the corresponding segment further includes deleting, by the data processing hardware, the corresponding segment of the audio data after determining the transcript.

3. The method according to claim 1, wherein: Applying the privacy condition to the corresponding segment further includes enhancing, by the data processing hardware, the corresponding segment of the image data to visually conceal an identity of the speaker of the corresponding segment of the audio data.

4. The method according to claim 1, wherein: Processing the plurality of segments of the audio data to determine a transcript of the audio data further includes modifying, by the data processing hardware, the transcript to exclude the identity of the speaker.

5. The method according to claim 1, wherein: Each other segment of the plurality of segments except the corresponding segment includes a corresponding voice utterance that is not spoken by the participant associated with the privacy condition indicated by the received privacy request.

6. The method according to claim 1, wherein: The privacy condition includes a content-specific condition indicating a type of content to be excluded from the transcription, and wherein omitting transcribing the corresponding segment is further based on the content-specific condition.

7. The method according to claim 1, wherein: Determining the identity of a speaker of the corresponding segment of the audio data from among the plurality of participants comprises: determining a plurality of candidate identities of the speaker based on the image data; For each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of the corresponding candidate identity based on the image data includes a speaking face of the corresponding segment of the audio data; and The identity of the speaker of the corresponding segment of the audio data is selected as the candidate identity among the plurality of candidate identities associated with a highest confidence score.

8. The method according to claim 1, wherein: The data processing hardware resides on a device local to at least one of the plurality of participants.

9. The method according to claim 1, wherein: The image data includes high definition video processed by the data processing hardware.

10. The method according to any one of claims 1 to 9, wherein: Processing the plurality of segments of the audio data to determine the transcript of the audio data further includes processing the image data to determine the transcript.

11. A system for privacy-aware transcription, comprising: Data processing hardware; as well as Memory hardware, the memory hardware being in communication with the data processing hardware, the memory hardware storing instructions, the instructions, when executed on the data processing hardware, causing the data processing hardware to perform operations, the operations comprising: receiving an audiovisual signal comprising audio data corresponding to voice utterances from a plurality of participants in a voice environment and image data representing faces of the plurality of participants in the voice environment; receiving a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the voice environment; Splitting the audio data into a plurality of segments; For each segment of the audio data: determining, based on the image data, an identity of a speaker of a corresponding segment of the audio data from among the plurality of participants; determining whether the identities of the speakers of the corresponding segment include the participants associated with the privacy condition indicated by the received privacy request; and When the identity of the speaker of the corresponding segment includes the participant, applying the privacy condition indicated by the received privacy request to the corresponding segment, wherein applying the privacy condition indicated by the received privacy request to the corresponding segment includes omitting transcribing the corresponding segment based on the privacy condition; and processing the multiple segments of the audio data to determine a transcript of the audio data, wherein the transcript includes text corresponding to each other segment of the multiple segments except the corresponding segment and omitting text corresponding to the corresponding segment.

12. The system according to claim 11, wherein: Applying the privacy condition to the corresponding segment further includes deleting, by the data processing hardware, the corresponding segment of the audio data after determining the transcript.

13. The system according to claim 11, wherein: Applying the privacy condition to the corresponding segment further includes enhancing, by the data processing hardware, the corresponding segment of the image data to visually conceal an identity of the speaker of the corresponding segment of the audio data.

14. The system according to claim 11, wherein: Processing the plurality of segments of the audio data to determine a transcript of the audio data further includes modifying, by the data processing hardware, the transcript to exclude the identity of the speaker.

15. The system according to claim 11, wherein: Each other segment of the plurality of segments except the corresponding segment includes a corresponding voice utterance that is not spoken by the participant associated with the privacy condition indicated by the received privacy request.

16. The system of claim 11, wherein: The privacy condition includes a content-specific condition indicating a type of content to be excluded from the transcription, and wherein omitting transcribing the corresponding segment is further based on the content-specific condition.

17. The system of claim 11, wherein: The operation of determining the identity of the speaker of the corresponding segment of the audio data from the plurality of participants comprises: determining a plurality of candidate identities of the speaker based on the image data; For each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of the corresponding candidate identity based on the image data includes a speaking face of the corresponding segment of the audio data; and The identity of the speaker of the corresponding segment of the audio data is selected as the candidate identity among the plurality of candidate identities associated with a highest confidence score.

18. The system of claim 11, wherein: The data processing hardware resides on a device local to at least one of the plurality of participants.

19. The system of claim 11, wherein: The image data includes high definition video processed by the data processing hardware.

20. The system according to any one of claims 11 to 19, wherein: Processing the plurality of segments of the audio data to determine the transcript of the audio data further includes processing the image data to determine the transcript.

Images