Handling overlapping speech from distributed devices

Abstract

A computer-implemented method comprising: receiving audio signals representing speech via multiple audio streams transmitted from a corresponding plurality of distributed devices; performing continuous speech separation for one or more of the received audio signals having overlapping speech via a neural network model; and providing the separated speech on a fixed number of separate output audio channels.

Description

Background Art

[0001] Meetings planned in advance may utilize one or more conferencing tools that are set up before or at the start of the meeting to record conversations and generate transcripts attributed to the speakers. Such existing conferencing tools may include a device with multiple fixed speakers on different sides of the device located on a conference table. The device may have a tower or cone shape and may have a video camera that can be used to identify and track people in the meeting. Speech-to-text algorithms may be used to create transcripts. Audio beamforming may be used with the known locations of the fixed speakers and the video of the attendees to attribute speech in the transcript. Summary of the invention

[0002] A computer-implemented method comprising: receiving audio signals representing speech via multiple audio streams transmitted from a corresponding plurality of distributed devices; performing continuous speech separation for one or more of the received audio signals having overlapping speech via a neural network model; and providing the separated speech on a fixed number of separate output audio channels.

[0003] Continuous speech separation may be performed by a neural network model trained using permutation invariant training. The plurality of distributed devices include wireless devices associated with speakers in a conference. The conference may be an ad-hoc conference. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a perspective view of a conference between multiple users according to an example embodiment.

[0005] Figure 2 is a block diagram of a user device for use in a conference according to an example embodiment.

[0006] Figure 3 is a flow chart illustrating a method of initiating a smart conference between two users having associated distributed devices, according to an example embodiment.

[0007] Figure 4 is a flow chart illustrating a method of adding distributed devices to a smart conference by using conference codes according to an example embodiment.

[0008] Figure 5 A computer-implemented method of adding additional devices to a smart conference according to an example embodiment.

[0009] Figure 6 is a flow chart illustrating a computer-implemented method of detecting that an ad-hoc conference is ongoing, according to an example embodiment.

[0010] Figure 7is a flow chart illustrating a computer-implemented method of removing audio channels from a user device and other devices in response to a user leaving a meeting, according to an example embodiment.

[0011] Figure 8 is a flow chart illustrating a computer-implemented method of authenticating a device for adding an audio stream from the device to an audio channel processed by a conference server instance, according to an example embodiment.

[0012] Fig. 9 is a high-level block flow diagram of a system for generating a transcript for a conference between multiple users according to an example embodiment.

[0013] Fig.10 is a detailed block flow diagram illustrating distributed conference server processing of information including audio streams from distributed devices according to an example embodiment.

[0014] Fig.11 is a flow chart illustrating a computer-implemented method of synchronizing multiple audio channels received from multiple distributed devices during an intelligent conference, according to an example embodiment.

[0015] Fig.12 is a flow chart illustrating a computer-implemented method of separating overlapping speech in a distributed device intelligence conference according to an example embodiment.

[0016] Fig.13 is a flow chart illustrating a computer-implemented method 13 of blending audio streams at a plurality of selected points during processing, according to an example embodiment.

[0017] Fig.14A and Fig. 14B An example environment capture device according to an example embodiment is illustrated.

[0018] Fig.15 Illustrated are example placements of microphone arrays according to example embodiments.

[0019] Fig.16 An AI system with an environment capture device according to an example embodiment is illustrated.

[0020] Fig.17 is a flow chart illustrating a computer-implemented method of reducing the number of audio streams sent over a network to a conference server for generating a transcript, according to an example embodiment.

[0021] Fig.18 is a flow chart illustrating a computer-implemented method for using video and audio channels, both audiovisual data, from distributed devices to provide better speaker identification, according to an example embodiment.

[0022] Fig.19 is a flow chart illustrating a computer-implemented method for customizing output based on user preferences, according to an example embodiment.

[0023] Fig. 20 is a schematic block diagram of a computer system for implementing one or more example embodiments. DETAILED DESCRIPTION

[0024] In the following description, reference is made to the accompanying drawings which form a part thereof, and in which specific embodiments that can be practiced are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it should be understood that other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. Therefore, the description of the following example embodiments should not be considered restrictive, and the scope of the invention is defined by the appended claims.

[0025] In one embodiment, the functions or algorithms described herein can be implemented in software. The software may include computer executable instructions stored on a computer readable medium or a computer readable storage device, such as one or more non-transient memories or other types of hardware-based storage devices (local or networked). In addition, this function corresponds to a module, which can be software, hardware, firmware, or any combination thereof. Multiple functions can be performed in one or more modules as needed, and the described embodiments are only examples. The software can be executed on a digital signal processor, an ASIC, a microprocessor, or other types of processors operating on a computer system such as a personal computer, a server, or other computer system, thereby converting such a computer system into a specially programmed machine.

[0026] Functions can be configured to perform operations using, for example, software, hardware, firmware, etc. For example, the phrase "configured to" can refer to the logical circuit structure of a hardware element for implementing the relevant functions. The phrase "configured to" can also refer to the logical circuit structure of a hardware element for implementing the coding design of the relevant functions of firmware or software. The term "module" refers to a structural element that can be implemented using any appropriate hardware (e.g., processor, etc.), software (e.g., application, etc.), firmware, or any combination of hardware, software and / or firmware. The term "logic" covers any function for performing a task. For example, each operation shown in the flowchart corresponds to the logic for performing the operation. The operation can be performed using software, hardware, firmware, etc. The terms "component", "system", etc. can refer to computer-related entities, hardware, and execution software, firmware, or a combination thereof. A component can be a process running on a processor, an object, an executable file, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term "processor" can refer to a hardware component, such as a processing unit of a computer system.

[0027] In addition, the claimed subject matter can be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof for controlling a computing device to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to cover a computer program accessible from any computer-readable storage device or medium. Computer-readable storage media may include, but are not limited to, magnetic storage devices such as hard disks, floppy disks, magnetic strips, optical disks, compact disks (CDs), digital versatile disks (DVDs), smart cards, flash memory devices, and the like. In contrast, computer-readable media (i.e., not storage media) may additionally include communication media, such as transmission media for wireless signals, and the like.

[0028] Individuals, referred to as users, can start a conversation or conference at any time. If a meeting has been planned, arrangements can be made to record the conversation and create a transcript of the conversation for later reference. However, ad-hoc meetings typically do not involve this kind of preparation. Stopping the meeting, or otherwise taking the time to set up a method to record the conversation and arrange for the creation of a transcript can be distracting or may not be thought of during the meeting. In addition, ad-hoc meetings are often held outside of a conference room. In this case, recording equipment designed for meetings is not available.

[0029] During the conversation, the audio of the conversation can be captured by a device that the user may carry, which is called a distributed device. The captured audio signal can be transmitted to the conference system through a wireless channel to identify that multiple users are having a conversation (called a conference), which may or may not be planned. If the conference is not planned, it can be called an ad-hoc conference.

[0030] In response to a meeting that has been detected or otherwise scheduled, a meeting instance is generated on a conference system to recognize speech from a user who may be speaking and to generate a transcript of the meeting. Multiple speech signals from multiple distributed devices may be received as separate audio channels and used to generate the transcript. The distributed devices may include personal user devices as well as other devices, including digital assistants, cameras, and any type of device capable of receiving audio and / or video that may be within range of a conversation.

[0031] In some embodiments, a conference can be created with a single button press on a single device via a conference application. Other devices and users with devices can join the conference by pressing a button presented on the user's device via the conference application or being recruited when not in use (e.g., an existing conference device present in the room). Conference participants can be inferred by voice fingerprinting, the owner of the participating device, facial recognition, and manually adding users via the conference application on their device at any point in time (e.g., for remote participants).

[0032] There are many different ways to set up a conference. Distributed devices such as smart phones can be associated with respective users and can include a conference application for streaming audio and / or video received from a microphone and / or camera on the device to a conference server. The audio received from a nearby device will have an audio signature that is based on a combination of ambient noise and / or any sound generated near the device. In response to two user devices providing similar audio signatures via their respective audio streams (audio channels), the conference system identifies that a conference may be ongoing and creates a conference instance to process the received audio. Users can be prompted to join the conference via their conference application, or other information, such as location information, previous interactions, calendar information, or recent email interactions, can be used to confirm that both users or a third user should be added to the conference instance.

[0033] In another embodiment, an audio watermark may be generated by one or more of the user devices. The audio watermark may include an audio signature or may be detected separately. The audio watermark may be a sound pattern with a frequency above the user's normal hearing range, such as 20Khz or higher, or may simply be a sound that is inconspicuous to the user so as not to interfere with the conversation. In another embodiment, the watermark may be fully audible and recognizable. In some embodiments, the watermark may be selected to be sent by a user who is desired to ensure that a conference instance is created during a conversation. The watermark will be received by distributed devices within range and automatically or optionally added to the conference instance. Devices within range of the watermarked sound may also add their audio stream as an additional audio channel to the conference instance.

[0034] Conference codes can be generated and sent to other users to add them to a planned or ad-hoc meeting. Conference codes can also be selected before a planned meeting and used in a meeting invitation. After receiving the conference code from a user device, the conference system, once instantiated, adds the audio stream from such user device to the meeting. In another embodiment, conference codes can be used to join multiple clusters of distributed devices. Clusters can be in different locations, such as different halls, rooms, cities, countries, etc.

[0035] In one embodiment, the output can be customized based on the user preferences in the distributed system. In an example embodiment, a conference server or system receives an audio stream from multiple distributed devices involved in a smart conference. A smart conference is just a conference in which one or more devices are used to capture voice and create a transcript that can have a speaker attribution. The conference system identifies the user corresponding to a distributed device in multiple distributed devices and determines the user's preferred language. As the conference proceeds, a transcript is generated from the received audio stream. The conference system translates the transcript into the user's preferred language to form a translated transcript. The translated transcript is provided to the user's distributed device. In an example embodiment, the translated transcript is provided in real time (or near real time) while the conference is in progress. The translated transcript can be provided via text (e.g., displayed on the user's device) or output as audio (e.g., via a speaker, hearing aid, earpiece). In some embodiments, as a substitute or supplement for translation, other types of conversions can be applied to the original transcript, the translated transcript, or the translated voice audio.

[0036] Figure 1 1 is a perspective view of a conference 100 between multiple users. A first user 110 has a first device 115 that includes a microphone to capture audio including speech. A second user 120 has a second device 125 that is also capable of capturing audio including speech. In an example conference 100, users may sit at a table 130.

[0037] The first device 115 and the second device 125 transmit the captured audio to the conference server 135 for processing and generating a transcript. The meeting can be ad-hoc in that it is not planned. The users may have run into each other during a break or happened to meet in the hallway and decided to talk about the project they are working on. A conference application or conference app can be run on both the first device and the second device. The conference application can be used to provide audio to the conference server 135.

[0038] Conference server 135 may detect that both devices are transmitting audio with similar audio signatures, audio watermarks, conference codes provided by both devices, or other information indicating an ongoing discussion between users. Conference server 135 may generate a conference instance to process the received audio and generate a transcript.

[0039] In various embodiments, the watermark may be any type of sound having energy just above the human hearing range (which is approximately 20 kHz), or an otherwise inaudible, unobtrusive, or non-distracting sound that identifies the meeting instance or meeting 100 corresponding to the meeting code. In another embodiment, the watermark may be a sound that encodes the meeting code or other identification of the meeting instance.

[0040] Conference 100 may involve more than two persons, whether planned or ad-hoc. A third user 140 with a third device 145 may also join conference 100. The third device 145 may also provide audio to distributed conference server 135. The audio may be identified as participating in conference 100 by one or more of the same mechanisms described for identifying the first two users / devices participating in conference 100.

[0041] The owner / user of the distributed device can add him / herself via the application to be recognized by the conference server 135. The user may already have a voice profile (called a voice print or fingerprint) to help the conference server 135 associate incoming voice with the user. If a random person joins the meeting 100, the conference server can recognize that the person is unknown and prompt one or more of the users already in the meeting to enter the name of the person, or it can also search a database in the organization associated with known users in the meeting to match the person with the profile. If the person is unknown or otherwise identified, the person can be identified as speaker 1, speaker 2, etc. in the generated transcript using a label or tag, making it easier to modify the transcript when the person is named later. Any user can assign a name to the speaker tag at any time during or after the meeting. The known contacts of those who are already in the meeting can be used to reduce the pool / database used for initial inspection of personnel to optimize the process of identifying personnel.

[0042] There may be additional devices within audio or visual range of the conference 100, such as a digital assistant 148 or a dedicated conference device 150, both of which are shown on the table 130, but may be anywhere within audio range of the conference 100. Such additional devices may also connect to the distributed conference server 135 and add their audio streams to the conference instance for processing, further enhancing the audio and speech-to-text processing capabilities of the conference instance running on the conference server 135. Such additional devices may be detected by the server and added to the conference as described above, or may be presented to one or more of the users as an option for adding to the conference.

[0043] Video camera 155 or other camera may have a field of view that surrounds conference 100. Conference server 135 may be aware that such a camera is in the vicinity of conference 100 and may provide an indication to one or more of the users, which provides the option of obtaining information from the camera into the conference instance to further enhance the processing and provision of the transcript. For example, such a camera may be used to detect which user is speaking, or at least provide information about which user may be speaking at any particular point in time.

[0044] Figure 2 is a block diagram of a user device 200 for use in a conference. Other devices that may participate in a conference may have a similar set of components. The device 200 includes at least one microphone 210 and a processor 215 for executing a conference application 220 stored on a memory 225. The transceiver 230 is used to stream audio and / or video from a camera 235 to a distributed conference server 135. The user device 200 may also have a display screen, such as a touch screen 240, a portion of which is shown.

[0045] Devices that can participate in a meeting can be identified by: calendar entry, current location, NFC (when the phone is brought very close together), Bluetooth Advertisements, and direct invitations via conference codes or other codes that may be generated and associated with conference 100 .

[0046] The conference server 135 can handle multiple conferences simultaneously via multiple conference instances. Each conference instance can include a conference identifier, such as a conference code, an identifier of a device streaming audio, an identifier of a user participating in the conference (which identifier is recognized by the conference server via a device associated with the user, or otherwise recognized by the conference server through facial recognition, voice recognition, or other means of identifying a user).

[0047] Figure 33 is a flow chart illustrating a method 300 of initiating a smart conference between two users with associated distributed devices. At operation 310, an audio watermark is received at the first distributed device via a microphone associated with the first distributed device. The audio watermark may be transmitted by a speaker associated with a second distributed device during the conference. At operation 320, data corresponding to the received audio watermark is transmitted to a distributed device conference server via the first distributed device. Note that the received audio watermark may first be converted to digital form, which may simply be a direct conversion of the audio watermark to a digital representation of the sound, or may include decoding of the audio watermark to obtain data identifying the conference or the second distributed device that issued the audio watermark. At operation 330, an indication may be received from the distributed conference server that the first distributed device has been accepted into a conference instance on the distributed device conference server. At operation 340, the first distributed device, in response to the received indication, streams the audio of the conference to the conference instance on the distributed device conference server. The received indication may include information identifying the communication channel to be used, or the audio stream may simply identify the streaming device, which the conference server uses to direct the audio stream to the correct conference instance.

[0048] Figure 4 4 is a flowchart illustrating a method 400 of adding distributed devices to a smart conference using a conference code. In some embodiments, the conference code can be encoded in a watermark as in method 300. At operation 410, a conference code is generated for a conference between users via a first distributed user device. The first distributed user device can receive the code from a conference server executing a conference instance, or can generate the conference code via a conference application running on the first distributed user device. At operation 420, the code can be sent to a second distributed user device. The code can be sent via email, text, or other electronic means of sending data, or can be encoded as an audible signal (audio watermark) and acoustically transmitted to the remaining participating devices, such as via a speaker of one of the user devices (such as the first distributed user device). The second distributed user can provide a conference code to the conference server conference instance, where the conference code is used at operation 430 to identify at least one second distributed user device. At operation 440, the second distributed user device streams audio from both the first and second distributed user devices to the conference server conference instance.

[0049] The conference can be an ad-hoc conference between multiple users with multiple corresponding user devices, and the conference code is generated after the ad-hoc conference starts. Note that there may also be users without associated user devices participating in the conference. Other user devices and devices not associated with the user can be identified based on the detected device location. Data from such devices can have their data streams added to the conference instance by providing the user with a list of other nearby devices and allowing such devices to be selected to be added to the conference instance via the user interface of the application. Devices that can participate in the conference can be identified via calendar entries, current location, NFC (bringing the phone close together), Bluetooth advertisements, and direct invitations.

[0050] In another embodiment, the conference is a planned meeting between multiple users with multiple corresponding user devices, and the conference code is generated before the planned meeting starts. The conference code can be sent to each user device in the user devices and can be used by the corresponding application to identify the device to the conference server conference instance to add data streams from these devices during the meeting.

[0051] Figure 5 The present invention is a computer-implemented method 500 for adding additional devices to a smart conference. At operation 510, a conference server receives an audio stream from a distributed device group, wherein the audio stream includes speech captured or detected by such distributed device group during a conference of more than two users. At operation 520, the conference server receives conference information corresponding to the conference from the additional or new distributed device. The new device can be a user device in the case where a user has just joined the conference, or the new device can be a device in the room or otherwise within the scope of the smart conference. At operation 530, the additional distributed device is added to the conference server conference instance. At operation 540, in response to adding the additional distributed device, a stream of information from the additional distributed device is received.

[0052] Figure 6 6 is a flow chart illustrating a computer-implemented method 600 for detecting that an ad-hoc conference is ongoing. At operation 610, audio streams are received at a conference server from two distributed devices that stream audio detected during an ad-hoc conference between two users. At operation 620, the audio streams are compared to determine that the audio streams represent sounds from the ad-hoc conference. At operation 630, in response to determining that the audio streams represent sounds from the ad-hoc conference, the conference server generates a conference instance to process the audio streams. At operation 640, the audio streams are processed to generate a transcript of the ad-hoc conference.

[0053] Many different algorithms for comparing two signals can be used to compare audio streams. In one embodiment, the comparison is performed by calculating the normalized cross-correlation coefficient between the two signals. If the result is above a selected threshold, the audio stream is likely to be from the same (ad-hoc) conference. The selected threshold can be a number between 0 and 1, and can be selected empirically based on tests conducted during multiple conference scenarios in different environments. The selection can be performed to obtain a desired balance of false negatives and false positives. Other indications that the stream is from the same conference include the location of the device. Additional indications include multiple users with multiple contacts in the past in the same organization, as well as other indications that the user may have encountered. Further verification can be obtained by comparing the text generated from the audio stream.

[0054] Once the streams are successfully compared, a conference id (identifier) / code can be generated and used to add more participants. In response to another device streaming audio that is successfully compared to the audio stream already in the conference, other participants can be added. Once a device is added, the device can generate a signal indicating joining the conference, such as a ping.

[0055] Users may also be authenticated before audio streams from their respective devices are added to the conference instance. Authentication may be based on user confirmation from an application, calendar information, an organizational chart, use of a conference code, degree of connection / relationship with users already in the conference, and other authentication methods.

[0056] In one embodiment, conference server 135 detects the time when the device and / or associated user leaves the meeting, and removes the audio stream / channel from the meeting instance. When the participant associated with the device leaves the meeting, conference server 135 detects the absence of the audio signal associated with other devices in the meeting, and removes the device from the meeting. Alternative solutions include the user signaling to leave via the conference application, closing the conference application, detecting that the location of the device is no longer close to the location of the meeting, detecting that there is no corresponding audio watermark in the video stream from the device, detecting that the audio signature received by the device no longer matches the audio stream of other devices, and performing image recognition on the image from the video signal to detect that the user is leaving or has left the conference room or area where the meeting is being conducted. Similarly, a conference instance can be ended in response to a remaining single user or a remaining single user device.

[0057] Figure 77 is a flow chart illustrating a computer-implemented method 700 for removing audio channels from a user device and other devices in response to a user leaving a conference. At operation 710, a plurality of audio signals are processed by a conference server instance, the plurality of audio signals being received on a corresponding plurality of audio channels received from a distributed device group that receives audio from a distributed device conference. At operation 720, the conference server instance is operable to detect that a first user associated with a first device of the distributed device group has left the distributed device conference. At operation 730, an audio channel of the first distributed device is removed from the plurality of audio channels processed by the conference server instance.

[0058] Figure 8 8 is a flowchart illustrating a computer-implemented method 800 for authenticating a device for adding an audio stream from the device to an audio channel processed by a conference server instance. The method 800 begins at operation 810, where an audio stream is received from a plurality of distributed devices at a conference server, which receive speech from a plurality of users during a conference. At operation 820, the received audio stream is processed via a conference instance executed on a distributed device conference server to generate a transcript based on the speech included in the audio stream. At operation 830, information is received at the conference server from a first additional distributed device associated with a first additional user, the information corresponding to a conference between the users. The information may correspond to a request to add a user's device, or may be an implied request (by noting that an audio stream from such a device includes a watermark or an audio signature). At operation 840, the first additional distributed device or associated user is authenticated or otherwise authorized to join the conference. At operation 850, the first additional distributed device adds its audio stream to the conference instance.

[0059] Participants may be authorized to join the meeting based on voice fingerprint, meeting organizer acceptance, use of conference code and / or new code, detected device location, comparison of device ID and / or associated user ID to an authorization list, organization membership check, use of a private meeting flag to require organizer acceptance, or a combination of the above. Note that method 800 may also be applied to the first two devices joining the meeting, and may also be applied to devices that are not directly associated with the user, such as a conference assistant type device in a conference room or a video camera with a field of view of the meeting.

[0060] Remote participants can connect to the meeting via Skype or Teams, or dial-in or any other teleconferencing application. If using a remote conferencing platform like Skype, the meeting can be joined by following a link sent in advance. For dial-in, a unique phone number or access code, such as a conference code, can be shared. Once the remote audio channel is connected to the meeting's server, it is treated in a manner similar to the audio stream from the conference area. If the speaker ID is known based on the login process, the audio stream can be a single speaker, which means that voice separation is not required unless a speakerphone is used with multiple remote users. In a meeting, audio played by the speakerphone and captured by nearby distributed devices should be eliminated from the audio stream from such nearby distributed devices.

[0061] Fig. 9 9 is a high-level block flow diagram of a system 900 for generating transcripts for a conference for multiple users. A user may have associated devices 910, 912, 914 equipped with microphones to capture audio, including the voices of various users in the conference, and provide the captured audio as audio signals via audio channels 916, 918, and 920, respectively, to a conference server, which includes at least a conference transcriber 925. Different devices may have slightly different clock cycles and different amounts of processing latency. In addition, each device-to-server connection channel may have a different latency. Therefore, the signals from audio channels 916, 918, and 920 are not necessarily synchronized.

[0062] In addition to the speech recognition function, the conference transcriber 925 may also include a synchronization function. The audio signals from channels 916, 918, and 920 are first synchronized and then recognized, which in one embodiment produces text associated with each of the channels. The recognition output is then fused or otherwise processed to generate a transcript 940. The transcript 940 may then be provided back to the user for viewing. In other embodiments, the audio signals from channels 916, 918, and 920 are fused before speech recognition. The audio signals acquired after fusion are recognized, producing a single text for use in the transcript. In some embodiments, the transcript may be provided with very little delay.

[0063] In various embodiments, the conversion of audio signals to text for use in conjunction with speaker identification is provided by the conference server 135, and the generation of transcripts that are log-formatted to identify the speakers. The functions performed by the server include synchronization, recognition, fusion, and logging functions. Although such functions are Fig. 9 The functions may be performed in varying orders in different embodiments. Fusion may be performed before identification, and may also be performed at various other points as described below.

[0064] Fig.10 1000 is a block flow diagram illustrating the processing of information (including audio streams from distributed devices) by a conference server, which receives multiple audio data streams 1005 from multiple distributed devices. The streams include M independent sequences of data packets. Each packet of the mth sequence contains a segment of a digitized audio signal captured by the mth device. The received packets are unpacked and the data from the packets are reassembled to create a multi-channel signal. The multi-channel signal can be represented as: {[x0(t), ..., x M-1 (t)]; t=0,1,…}.

[0065] Synchronize the audio channels:

[0066] The digitized signals of different channels in a multi-channel signal are likely to be out of sync because many of the distributed devices are subject to differences in digital signal processing, differences in software latency on the devices, and differences in signal transmission speeds. All of these differences can add up, making it difficult to integrate information from different devices to create an accurate transcript. The stream synchronization module 1015 receives a multi-channel signal and selects one of the channels as a reference channel. Without loss of generality, the first channel can be used as the reference channel. For the reference channel, the output is the same as the input, i.e., y0(t) = x0(t). For the mth channel (0 <m<M),x m The misalignment between x(t) and x0(t) is estimated and corrected to generate y m (t).

[0067] The degree of misalignment can be estimated by calculating the normalized cross-correlation coefficient between the two signals, using a sliding window of the non-reference channel signal and picking the lag that provides the maximum coefficient value. This can be achieved by using a buffer to temporarily store acoustic signal segments on which the cross-correlation analysis is performed separately between the reference channel and each of the other channels. Instead of the normalized cross-correlation, any score function that measures the degree of alignment between the two signals can be used.

[0068] In one embodiment, the relationship between adjacent synchronization cycles is considered. Misalignment is caused by two factors: device / channel-related offset and device-related clock drift. Even if two devices capture sound events at the same time, the signals captured by separate devices arrive at the server at different times due to DSP differences, differences in software latency on the device, differences in signal transmission speed, etc. This is device / channel-related offset. In addition, due to manufacturing variability, different devices inevitably have slightly different clocks. Therefore, even if two devices claim to support, for example, a 16kHz sampling rate, the signals recorded by these devices are not 100% aligned, and the amount of misalignment grows linearly over time. This is device-related clock drift. Device / channel-related offset and device-related clock drift are denoted as S and D. The time difference of the kth synchronization cycle is denoted as S+kD. Therefore, an estimate of S and D will provide a robust estimate of the degree of misalignment S+kD.

[0069] The amount of misalignment can be corrected by periodically detecting misalignment using the above-mentioned cross-correlation and correcting such detected misalignment. In addition, in order to reduce the measured amount of misalignment, a global offset (device / channel related) and a device-related clock drift can be calculated to estimate the degree of misalignment. The global offset can be used to correct the global misalignment before measuring and correcting the misalignment by cross-correlation. The global offset can be determined as the average of the measured misalignment over time and is likely the result of clock drift in the device. Therefore, the degree of misalignment is estimated and corrected by simply considering the difference from the reference channel. Stream synchronization can be performed at varying intervals, such as every 30 seconds. In another embodiment, other intervals less than or greater than 30 seconds can be used because network delays can vary.

[0070] The stream synchronization module 1015 converts the multi-channel synchronization signal {[y0(t),…,y M-1 The beamforming module 1020 is used to separate overlapping speech. Overlapping speech occurs when two people in a meeting speak at the same time. Before the speech is recognized and converted to text, the speech is first separated into separate channels. Therefore, for an M-channel input, the output is N-channel and is referred to as an N-channel beamforming signal {[z0(t),…,z N-1 (t)]; t=0,1,...}. The stream synchronization module 1015 acts as the first fusion point where multiple outputs are generated to maintain the diversity of the input information. In the absence of speech overlap, this fusion is optional.

[0071] Fig.111 is a flow chart illustrating a computer-implemented method 1100 for synchronizing multiple audio channels received from multiple distributed devices during a smart conference. At operation 1110, audio signals representing streaming voice are received from multiple distributed devices to generate multiple audio channels. At operation 1120, a selected audio channel of the audio channels is designated as a reference channel. Once the reference channel is designated, the following operations are performed for each of the remaining audio channels. At operation 1130, a time difference with the reference channel is determined. At operation 1140, the time of each remaining audio channel is aligned or compensated by correcting the remaining audio channels using the reference channel according to the corresponding time difference. This can be done by simply discarding irrelevant samples, shifting signals, appending zeros, or using resampling techniques.

[0072] Method 1100 can be performed periodically (such as every 30 seconds) to correct the timing of the remaining audio channels. In one embodiment, method 1100 includes other operations for correcting the global offset caused by at least different clocks in the distributed devices. At operation 1150, a global offset is determined for each audio channel in the remaining audio channels. Then, at operation 1160, before correcting each remaining audio channel for the determined time difference, the remaining audio channels are then corrected by each corresponding remaining audio channel global offset.

[0073] Geometry Agnostic Beamforming:

[0074] Acoustic beamforming, or beamforming for short, is a technique for enhancing target speech by reducing unwanted sounds, such as background noise, from a multi-channel audio signal. It can improve the accuracy of downstream speech processing, such as speech recognition and speaker logging.

[0075] For smart conferencing that streams audio from multiple distributed devices whose exact locations relative to each other are unknown, traditional beamforming algorithms such as delay-sum beamforming, super-directional beamforming, and differential beamforming do not work. Such algorithms rely on prior knowledge about the placement of microphone devices, which is not available to distributed devices.

[0076] In one embodiment, a method known as geometry-agnostic beamforming or blind beamforming is used to perform beamforming for distributed recording devices. Given M microphone devices corresponding to M audio channels, the M-dimensional spatial covariance matrix of speech and background noise is directly estimated. The matrices capture the spatial statistics of speech and noise, respectively. To form the sound beam, the M-dimensional spatial covariance matrix is ​​inverted.

[0077] A disadvantage of the beamforming method (whether traditional geometry-based beamforming or blind beamforming) is that it typically reduces the number of information streams from M to one, which means that downstream modules cannot take advantage of the acoustic diversity provided by spatially distributed devices. In order to generate M beamformed signals and maintain acoustic diversity, a leave-one-out approach can be used. Using this approach, a first output signal is generated by performing beamforming using microphones 2-M. A second output signal is generated using microphones 1-M and 3-M. This can be repeated M times so that M different output signals are obtained. For each beamforming, the (M-1)-dimensional spatial covariance matrix is ​​calculated and inverted, which is computationally demanding. Fortunately, the computational cost can be significantly reduced by deriving all (M-1)-dimensional inverse matrices from the original M-dimensional inverse matrix.

[0078] Separate overlapping speech:

[0079] In some embodiments, the beamforming module 1020 can be configured to separate overlapping speech signals of different users. This can make speech recognition and speaker attribution more accurate. In one embodiment, continuous speech separation for a distributed microphone recording system is performed via a neural network that is trained using permutation invariant training or a variation thereof, such as deep clustering or an attractor network. In order to potentially save computation, overlap detection can be used to determine whether a speech separation neural network should be executed for each time period. If overlapping speech is not detected for a selected time period, the neural network is not executed, which saves processing resources and allows for faster real-time generation of transcripts.

[0080] A speech separation neural network model is implemented to perform continuous speech separation for a distributed microphone recording system where the number of input microphones can be arbitrary and vary over time. The neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and the location of distributed devices during a conference between multiple users. The model outputs two continuous speech streams. When there is one active speaker, one of the output streams will be silent, while when there is overlapping speech between two speakers, each speaker will occupy a different output stream.

[0081] The speech separation neural network model contains three submodules: local observer, global aggregator, and mask reconstructor. Multi-channel inputs are processed by these three modules in sequence. First, the same local observer is applied to each input microphone. The local observer consists of a set of stacked attention layers, which maps each microphone input into a high-dimensional representation in which each channel is cross-compared and extracts information from all other channels. Two different types of attention are implemented, namely self-attention and feed-forward attention.

[0082] Next, an aggregator is applied to aggregate the information from each observer to form a global representation across different input channels. Two options for the aggregator are explored, which are mean pooling and a permutation-invariant ranking algorithm, where the representation of each channel is compared with a permutation-invariant loss to align their local and global permutations. When there is no aggregation layer, the network is reduced using a channel-by-channel speech separation network, where each channel has its own separation, i.e., there is no global separation agreement between channels.

[0083] Finally, for any time, the mask reconstructor simultaneously sorts the two mask outputs. The reconstructor consists of a stack of long short-term memory networks and generates the final 2-channel output from the summation at each time point.

[0084] After getting the two channel outputs from the mask reconstructor, a permutation invariant training objective function is applied between the reconstructed mask and the clean reference, where the Euclidean distance of each permuted output pair and the clean reference is first measured, and then the minimum distance and the corresponding permutation are selected to update the neural network.

[0085] The network was trained using simulated multi-channel data, where the number of input channels was randomly selected for each sample, from 2 channels to 10 channels. The Libri speech dataset was used as the source data in the simulation. In each simulated sentence, two utterances from two random speakers were first selected, and then each utterance was processed using room acoustic simulation, where the room impulse response came from an image method with a random room and position setting. Overlapping speech was generated by adding the two simulated speech with an expected overlap of 50%. 1000 hours of training data were simulated. The Adam algorithm (a replacement optimization algorithm for stochastic gradient descent for training deep learning models) was applied for updating the neural network.

[0086] A variation of speech separation is speech overlap detection, where the task is simplified to just detecting overlapping regions in the recorded speech. The algorithm operates in a similar manner, where the network receives N channels as input and outputs 2 channels consecutively as output. In the overlap detector, instead of outputting a mask, the network outputs two one-dimensional indicator functions, where 1 means there is an active speaker in that channel and 0 means silence. Therefore, when there are two active speakers, the 2-output streams will both have 1 as output. When there is one active speaker, either channel will have 1 as output and the other will have 0. The network is also trained using a permutation-invariant training objective between the output of the network (i.e., the indicator function) and the reference indicator.

[0087] Fig.121 is a flowchart illustrating a computer-implemented method for separating overlapping speech in a distributed device intelligent conference. At operation 1210, an audio signal representing speech is received via a plurality of audio channels corresponding to the streaming audio transmitted from the corresponding plurality of distributed devices. At operation 1220, continuous speech separation is performed on the received audio signal to separate the speech from different speakers who are speaking at the same time into separate audio channels. At operation 1230, the separated audio channels are provided for speech recognition and transcript generation. In one embodiment, operation 1230 provides a fixed number of separated output channels. Since there can be a variable number of microphone inputs, and the number of outputs is fixed in advance, there may be a situation where a limited number of audio channels can be accommodated, because for each audio channel with multiple overlapping speakers, each speaker produces a separated audio channel. Therefore, if the number of output audio channels is limited, not all channels can separate speakers.

[0088] In one embodiment, speech separation is performed by a trained neural network model at operation 1220. The neural network model is trained using permutation invariant training or a variation thereof.

[0089] Fig.10 The N different outputs of the beamforming module 1020 in are provided to N acoustic models 1025 and 1030, which generate a sequence of polyphonic senone posterior probabilities. Such models are well known and are typically based on neural networks. The acoustic model is used to provide N scores for each senone for each of the multiple audio channels from the distributed device and / or beamformer output.

[0090] The scores (including the scores for senones) are provided to the acoustic model score fusion module 1035. The audio of the individual input channels can be processed conventionally to provide a sequence of senones and their posterior probabilities. Before the results are applied to multiple speech recognition (SR) decoders 1040, 1045, the model score fusion module 1035 is used to combine the results. The score fusion module 1035 operates as a second fusion point, which combines multiple information sources and generates multiple outputs at the same time to retain the diversity of input information. The two-step process involves two different neural networks (or classifiers): a vanilla-flavor acoustic model and a new, more targeted acoustic model. The output is a sequence of 1x the number of senones. Note that the score fusion module 1035 uses the output of the last layer of the acoustic model (neural network) as input. In another embodiment, the score fusion module 1035 can use the output of any layer before the last layer. The size of the input may be different from the size of the output.

[0091] The sequence of senones from the acoustic model score fusion module 1035 is provided to SR decoders 1040 and 1045, each of which uses standard speech recognition processing to provide an n-best word list for each segment of the senone. A start time and duration are provided for each word. Segments can be executed based on voice activity detection, speaker change detection, fixed intervals, or some other appropriate method. Rescoring can be performed by using a neural network language model (NNLM) on the decoder output to generate a better n-best word hypothesis list.

[0092] Multiple speaker logging modules 1050, 1055 receive the output of the SR decoder module as an N-best list for each segment. In one embodiment, only the top word sequence hypotheses are used. The first operation extracts speaker embeddings, such as d-vectors (hidden layer activations of a deep neural network for speaker verification) at fixed intervals. The second operation decomposes the word sequence into sub-segments that are speaker isomorphic. This can be performed by a variation of agglomerative clustering, BIC (Bayesian Information Criterion), or other methods using embedding features. The third operation assigns a speaker ID to each of the sub-segments obtained above by comparing the proximity of the speaker embedding of the sub-segment to the embedding of each candidate speaker (e.g., cosine similarity, negative Euclidean distance). The output produced is a speaker label assigned to each recognized word of the top SR hypothesis.

[0093] The hypothesis combination module 1060 receives the n-best lists from N SR decoder modules 1040, 1045 (e.g., beamformed audio channels) and receives as input the speaker recognition outputs from N sources such as beamformed / separated audio channels. The hypothesis combination module 1060 processes the n-best scores from each channel by scaling and normalizing them and computing utterance-level posterior probabilities accordingly. The n-best hypotheses are aligned into a word confusion network. By adding the utterance-level posteriors associated with a given word hypothesis, word-level posterior probabilities can be obtained. The speaker recognition outputs from each channel are formatted as a confusion network with alternating speaker and word labels. The word labels come from the 1-best recognition hypothesis, while the speaker labels represent the 1-best or n-best speaker model that matches the speech segment. The posterior probabilities of the speaker hypotheses represent the likelihood of the normalized speaker model. The posteriors of word hypotheses are scaled down by two orders of magnitude to not affect the final word identification, and thus only the correct alignment of words and speaker labels. The confusion networks obtained from each channel are therefore truncated and / or concatenated as needed to cover the same time window, as dictated by the online processing constraints. The output consists of a confusion network (CN) that encodes the word and speaker hypotheses and their posterior probabilities.

[0094] The word and speaker confusion networks are aligned according to a minimum edit distance criterion and a penalty for temporal differences between aligned nodes. This effectively merges the speaker and word hypotheses into a single network, summing the posteriors for matching labels. If necessary, the top speaker and word hypotheses are read from the combined CN by picking the highest posteriors at each position. The word confusion network can be constructed from a word lattice instead of an n-best list, depending on the content of the speech decoder output.

[0095] The output from the combining module 1060 is the result of a third fusion, known as late fusion, to produce text and speaker identifications for generating a speaker-attributed transcript of the conference. Note that in various embodiments, the first two fusion steps at the beamforming module 1020 and the acoustic model score fusion module 1035, respectively, are optional. In some embodiments, one or more audio channels may be provided directly to the acoustic model scoring module 1065 without beamforming or speech separation. Speech recognition is then performed on such one or more audio channels via the SR decoder 1070, followed by the speaker logging module 1075, and the output is provided directly to the combining module 1060.

[0096] After synchronization of the digital audio streams, the audio streams can be early fused by geometry-agnostic beamforming or continuous speech separation. Multiple outputs can be generated to maintain input information diversity. Late fusion can be done at the acoustic model score level and / or text level / logging level to utilize speaker information and diverse model hypotheses. In one embodiment, late fusion of one or two words is performed by using a fixed time window. In one embodiment, the time window corresponds to a significant audio event and can be fixed at, for example, 2 seconds. This time window is selected to be quite short so that a real-time transcript with low latency can be provided.

[0097] Phrases are generated by performing late fusion of data on multiple audio channels processed in parallel through speech recognition. Phrases derived from multiple audio channels are combined in real time. In one embodiment, approximately two seconds of speech are combined at hypothesis combination module 1060. Therefore, audio streams are processed as they are received. A two-second non-overlapping sliding window is used to process the audio streams, thereby reducing the latency of conference server 135 transcript generation to near zero.

[0098] The individual speech recognition decoders continuously output some results, and based on the hypothesis combination module 1060, the results are processed immediately. At the stream synchronization module 1015, special provisions are provided for the alignment of the individual systems, otherwise the final result may contain multiple instances of the same event (due to misalignment). Regardless of the alignment of the signal and / or speech recognition output, the post-processing step removes any repetition that may exist. Alignment can be performed at the word line level or the sample level of the signal. Note in addition that different versions of the audio are received by the speech recognition decoder. Each SR decoder may have heard different things. By combining the SR results (late fusion) with low latency, a highly accurate transcript can be generated. Each SR outputs one or two words with confidence. A time such as two seconds is long enough to obtain some significant outputs, in other words, the output has one or two words that can be recognized with a certain degree of confidence. It is found that a fixed time window (such as two seconds) works better. If the time is too short, there is no significant event, and if the time is too long, the delay becomes too long, and the transcription is delayed, reducing the practicality of the transcript during the meeting.

[0099] Another version of this approach is to wait for points in the audio stream where (1) all streams contain no high-confidence speech, or (2) there are single-word hypotheses with high confidence. At these locations, the hypothesis space can be compressed to a single hypothesis, which allows the combination to be performed without losing accuracy due to word segmentation errors.

[0100] A transcript may be provided to one or more of the conference participants based on the output indicated at 1080. A single conference transcript is provided based on the output of the conferencing system. The transcript consists of individual utterances and associated media, such as slides or drawings. Each utterance is assigned a universal timestamp, an attributed speaker, associated text, and an associated audio segment, where the audio is extracted from synchronized input streams from all participating clients.

[0101] Additional media (such as images, notes, and other abstract objects) can be associated with the transcript inline via a timestamp (e.g., a picture of a whiteboard was captured and uploaded at time t), or associated with the entire meeting without a specific timestamp (e.g., a file was uploaded after the meeting and associated with that meeting instance). All participants have access to the meeting and associated data. Ad-hoc meetings can be viewed and modified by the meeting owner, all participants, or anyone, depending on the permissions set by the entity that created the meeting. Transcripts and other associated meeting data can be used to provide additional services such as meeting summaries, action item identification, and topic modeling.

[0102] Multi-level fusion

[0103] Fig.13It is a flowchart of a computer - implemented method 13 for fusing audio streams at multiple selection points during processing. Audio streams are recorded by multiple distributed devices during a meeting. Method 13 is executed by one or more processors performing operations. Operation 1310 performs speech recognition on each audio stream via a corresponding speech recognition system executed on one or more processors to generate utterance - level posterior probabilities as hypotheses for each audio stream. In operation 1320, the hypotheses are aligned and formatted into a word confusion network with word - related - level posterior probabilities. Operation 1330 performs speaker recognition on each audio stream by executing a speaker identification algorithm that generates a stream of word hypotheses for the attributed speaker. At operation 1340, for each audio stream, the speaker hypothesis is formatted into a confusion network together with the associated post - speaker - label posterior probability and the hypothesis of the attributed speaker. Operation 1350 aligns the word and speaker confusion networks from all audio streams with each other to combine the posterior probabilities and align the word and speaker labels. At operation 1360, the best - attributed speaker's word transcript is created by reading out or selecting the sequence of words and speaker labels with the highest posterior probability.

[0104] In one embodiment, when only a single word hypothesis is generated from each stream, there may even be no posterior probability, and a special approximate version is obtained when using simple voting among all streams.

[0105] The operations of method 1300 can be performed on consecutive time windows applied to the audio streams such that the processing is performed progressively to enable real - time generation of word recognition hypotheses for the attributed speaker. Based on the time stamps associated with the word hypotheses generated for each audio stream, the input hypotheses are truncated in time to a common time window applied to all audio streams.

[0106] Via the fusion of K out of N audio streams (where K < N), the input speaker and / or word hypothesis streams can originate from multiple partial combinations of the input audio streams. Alternatively, the input speaker and / or word hypothesis streams do not originate from different audio streams but from multiple partial combinations of acoustic models applied to K out of N audio streams, where the N audio streams can in turn originate from the original audio signal or a fusion of audio signals.

[0107] In yet another embodiment, based on the time notations associated with the word hypotheses generated for each audio stream, the input hypotheses are truncated in time to a common time window applied to all audio streams. The combination of K out of N original audio signals or the fusion of audio signals can be based on audio quality criteria and / or on the relative position of the speaker with respect to the distributed devices.

[0108] In one embodiment, via the fusion of K out of N audio streams (where K < N), the input speaker and / or word hypothesis stream is derived from a combination of multiple parts of the input audio stream. The combination of K out of N acoustic model outputs can be based on an audio quality criterion of the input signal and / or on the relative position of the speaker with respect to the distributed devices. Alternatively, the input speaker and / or word hypothesis stream can be derived from a combination of multiple parts of an acoustic model applied to K (where K < N) out of N audio streams, where the N audio streams are in turn generated from an original audio stream or a fusion of audio streams. In yet another embodiment, the outputs of multiple acoustic models can be applied to K (where K < N) out of N audio streams, where the N audio streams are in turn generated from an original audio stream or a fusion of audio streams, and these audio streams are combined as inputs to M speech recognition decoders.

[0109] Reducing computational resources in an ambient capture device:

[0110] Fig.14A and Fig. 14B Illustrates an example ambient capture device 1410. In one embodiment, the ambient capture device 1410 can be cylindrical, having a fisheye camera 1411 located at the top of the device and facing up with respect to the device. A microphone array 1413 can be coupled to the device 1410 below the camera 1411 and placed around the cylinder to capture audio 360°. It should be noted that Fig.14A the devices in [the figure] may not be drawn to scale. To capture optimal 360° vision (e.g., video or still images), it can be expected that the fisheye camera is close to the floor or table surface 1450. In one embodiment, the device can be shorter and short and wide to avoid blind spots below the camera 1411. In one embodiment, the fisheye camera can be placed adjacent to the microphone array 1413. In Fig. 14B the illustrated example, the microphone array 1413 includes seven microphones 1423A - 1423G. As shown, six microphones 1423A - 1423F can be placed around the device in a plane and more or less equidistant from the center of the device, and the seventh microphone 1423G can be placed at the center. It should be understood that the device can be made of audio - penetrable materials such as lightweight fabric, grille, or mesh, and the microphones 1423 will not be blocked by the fisheye camera 1421 or other structural parts of the device 1420, so that sound is not obstructed.

[0111] In one embodiment, the fisheye camera can be approximately 30 cm from the base of the device 1420, and the microphone array 1413 can be fixed approximately 15 cm above the base 1430. In operation, the device 1420 can sit or be attached to the floor or table 1450 in the environment. Because the device 1420 is placed closer to the floor, the 360° horizontal field of view (HFOV) can include more of the environment. The fisheye camera 1421 is typically attached to the device 1420 facing upward so that the ceiling can be within the field of view. It should be understood that other shapes, sizes, or configurations of the device 1420 and other placements of the fisheye camera 1421 and microphone array 1423 can be implemented, and some adjustments can be made to provide similar and different results.

[0112] In one embodiment, the acoustic parameters used for audio capture can vary depending on the specifications of the microphone. An example of the acoustic specifications of an embodiment is shown in Table 1 below. In one embodiment, the acoustic parameters (e.g., captured pulse code modulation (PCM) data) can be applied to the entire audio subsystem, not just the microphone. The captured audio can produce sufficient speech recognition accuracy for use in AI applications. Those of ordinary skill in the art who benefit from this disclosure should understand that various acoustic parameters can be used to achieve speech recognition accuracy, and the example parameters in Table 1 are for illustrative purposes.

[0113]

[0114]

[0115] Table 1. Example acoustic parameters

[0116] Fig.15An example placement of a microphone array 1523 according to an embodiment is illustrated. In one embodiment, the device includes seven microphones placed in the same plane. Six microphones 1523A-1523F can be placed in a circular or hexagonal pattern in a plane, approximately 4.25 cm from the center point. The seventh microphone 1523G can be placed at the center point. In one embodiment, the configuration of the seven microphones includes microphones of similar specifications. It should be understood that when the microphones are different, additional processing may be required to normalize or adjust the audio data received from the microphone array. In an example implementation, the microphone array 1523 may include seven digital micro-electromechanical systems (MEMS) microphones with ports facing upward. It should be understood that when the microphones are not blocked by sound absorbing components or sound blocking components (such as circuit boards or device housings), better performance can be produced. In one embodiment, similar microphones are timed using the same clock source (not shown) in the device. The timing or timestamp of the audio can help the synchronization and fusion of audiovisual data. The environment capture device can extract all microphone signals into 16-bit 16kHz PCM data. In this case, extraction is the process of reducing the sampling rate of the signal. For automatic speech recognition, a frequency band above 8kHz may not be required. Therefore, a sampling rate of 16kHz may be sufficient. Decimation reduces the bit rate without affecting the required accuracy. In one embodiment, the capture device may support additional bit depths and sampling frequencies. In one embodiment, the capture device may not allow changes to the data width and sampling frequency to reduce driver complexity and improve stability. The microphones may be mounted using any appropriate mechanical damping mechanism (e.g., rubber gaskets) to reduce vibration and noise. It should be understood that there may be more or fewer microphones in the microphone array. However, fewer microphones may introduce some uncertainty in the position or direction of the speaker. Additional microphones may provide increased audio certainty or resolution, but at the expense of more hardware and additional computational complexity.

[0117] In one embodiment, an audio speaker may be located at the bottom or base of the device for audio feedback to the user. The audio speaker may be used for feedback announcements or as an integral part of an AI application. For example, in an AI application for meeting management, a user may request that meeting minutes be read back to the attendees. The integrated speaker in the device may provide feedback or request instructions or commands for an operation. If a verbal command is not understood, a request to repeat the command may be played through the speaker. To reduce acoustic feedback, the audio speaker may face in the opposite direction of the microphone array. The audio played via the audio speaker may be fed back as an additional synchronous microphone channel.

[0118] Return to reference Fig. 14BIn one embodiment, the fisheye camera 1421 can receive a 360° HFOV and at least 95° vertical field of view (VFOV) above, and can receive 95° VFOV below the horizontal axis, resulting in a 190° VFOV or approximately 200° diagonal field of view (DFOV). In practice, the capture device can be placed on a table on the floor, so a vertical view below the surface may not be required. Therefore, in the discussion of this article, the VFOV is identified as approximately 95° to indicate a view above the horizontal base plane of the device. In one embodiment, the fisheye camera 1421 may include a 12 megapixel (MP) fisheye sensor (e.g., providing 4K resolution). The camera lens can be mounted relative to its image sensor so that the optical center is aligned with the center of the image sensor and the optical axis is perpendicular to the image sensor. The relative position of the camera module to the microphone array can be fixed and known. In particular, the optical center can also be aligned with the center of the microphone array, where the optical axis is perpendicular to the microphone array.

[0119] Fig.16An AI system 1600 with an environment capture device 1610 as described above and a conference server referred to as a cloud server 1620 is illustrated. In one example, a user 1630 interacts with an AI application 1623. It should be understood that the AI ​​application can reside on a cloud server 1620 or a local device (not shown). The AI ​​capture device 1610 can capture audiovisual data at 360°. As described above, the capture device 1610 may include a fisheye camera 1611 that provides a 360° HFOV and approximately a 95° VFOV. The capture device 1610 may include a microphone array 1613 to capture audio at 360°. Video compression of images and video streams received by the camera 1611 may be performed by a processor 1615 on the device. Video modes and compression protocols and standards may be controlled by user-selectable software controls. In addition to compression, audiovisual data may also be protected by encryption to prevent unauthorized persons from obtaining data. In one embodiment, compression 1618 may be performed by a circuit device on the device and controlled by a software switch. Preprocessing 1617 (e.g., cropping or noise reduction of an image based on image content) may be performed by logic executed by a processor prior to compression 1618. In one embodiment, preprocessing may include acoustic echo cancellation (AEC) to reduce feedback, noise, and echo caused by a speaker 1612 coupled to the device. In one embodiment, a local process for keyword spotting (KWS) may be included so that device commands for an environment capture device are listened for, such as to wake up or shut down the device. Local KWS may be beneficial for recall versus accuracy, and it may be based on a reduced microphone array (e.g., two microphones instead of a full array). When AEC is performed on the device 1610, the acoustic channel including the speaker audio may not need to be sent to the model to perform sensor fusion 1621. The compressed audiovisual data may be sent to a cloud server 1620 via a transmission unit 1619. The transmission unit 1619 may include one or more of the following: a network interface card for wired communication, such as an Ethernet connection; a wireless transceiver using a wireless protocol, such as NFC; or other communication methods. In one embodiment, the audio feedback can be sent to the device via one of the wireless channels. The cloud server 1620 can perform sensor fusion 1621 for the AI ​​application 1623. Therefore, compression can be performed to reduce the bandwidth transmitted to the cloud via the transmission unit 1619.

[0120] Capture device 1410 can be used with distributed devices to capture audio and video from a distributed device conference. Device 1410 itself can be one of the distributed devices. In one embodiment, identification of a user associated with speech can be performed solely by capture device 1410, and in various embodiments, information streams collected from capture device 1410 can be used together with information streams collected from other distributed devices to generate transcripts attributed to speakers.

[0121] Fig.17 17 is a flow chart illustrating a computer-implemented method 1700 for reducing the number of audio streams sent over a network to a conference server for generating a transcript. The method 1700 begins by receiving multiple audio channels from more than three microphones at operation 1710 that detect speech from a conference of multiple users. At operation 1720, the direction of the active speaker is estimated. At operation 1730, two channels are selected using a speech unmixing model, which may correspond to a primary and secondary microphone or may correspond to a fused audio channel. At operation 1740, the two selected channels are sent to the conference server to generate an intelligent conference transcript. By reducing the amount of data sent to the conference server, bandwidth can be saved. Since the selected data is arguably the best data, little accuracy, if any, is lost.

[0122] In one embodiment, the microphone is supported by a device in a fixed configuration. The fixed configuration may include a camera having a field of view configured to include multiple users. Localizing the sound source may be performed by executing a model trained on channels of audio and video from the camera. For example, if a user uses a laptop with a camera, the laptop may provide both an audio channel and a video channel. The audio channel may be synchronized relative to a reference audio channel, and the same time difference may be used to synchronize the video channel. Image recognition may be used on the video channel to identify the user as a speaker for logging when a transcript is generated. In another embodiment, the laptop may perform image processing to determine that the user is speaking, and provide a tag on the audio channel that identifies the user as a speaker and indicates whether the user is speaking. The tag may then be used for logging without transmitting the video channel from the laptop.

[0123] In another embodiment, the microphone is associated with a plurality of distributed devices. The distributed devices may include wireless devices respectively associated with a plurality of users. At least one of the distributed devices may include a camera that provides a video of at least one of the users.

[0124] In yet another embodiment, the microphone includes a microphone supported in a fixed configuration and includes a microphone associated with a distributed device associated with a user. The method can be performed by one or more edge devices of a device supporting the microphone in a fixed position or an edge device receiving multiple audio channels. The speech demixing model can be executed on the edge device.

[0125] In another embodiment, client-side processing (processing on one or more of the distributed devices, environment capture devices, and / or edge servers) is used to reduce the computing resources required by the conference server, and to reduce the amount of network bandwidth used to process distributed conference information streams from the distributed devices. In addition to reducing the number of streams sent to the conference server via the network as described above, beamforming can also be performed on the client side, as well as generating audio watermarks and conference codes. In another embodiment, the model size can be reduced and quantized to better run on the client side. The objective function can also be modified to better run on the client side. Instead of outputting a speech mask, sound source localization can be used with correspondingly less computation.

[0126] Both the audio channel and the video channel can be used to attribute speech to the user to create a log-based transcript. The audio-visual log-based approach allows combining speech identification, sound source localization, face tracking / identification, and visual active speaker detection from distributed sensors to achieve robust log-based.

[0127] Fig.18 1 is a flowchart illustrating a computer-implemented method 1800 for providing better speaker identification using video channels and audio channels, audio-visual data from distributed devices. The method 1800 begins at operation 1810, receiving information streams from a collection of multiple distributed devices included in a smart conference on a conference server. At operation 1820, an audio signal is received, the audio signal representing the speech of at least two users in at least two of the information streams. At operation 1830, at least one video signal of at least one user in the information stream is received. At operation 1840, based on the received audio and video signals, the received audio signal and video signal are used to associate the speech in the received audio signal with a specific user. At operation 1850, a transcript of the smart conference is generated, the transcript having an indication of the user associated with the speech.

[0128] In one embodiment, the plurality of distributed devices are mobile wireless devices associated with users in a smart conference. The mobile wireless device may include a microphone and may also include a camera that provides at least one video signal. In another embodiment, the plurality of distributed devices include a device having a plurality of microphones supported in a fixed configuration, each microphone providing one of the received audio signals. The device may include a camera having a field of view that is configured to include a plurality of users in the smart conference and provide at least one video signal.

[0129] In one embodiment, a fusion model is used on the received audio and video signals to associate a specific user with a voice. In one embodiment, the audio-visual data can be analyzed by a conference server. The audio-visual data can be first compressed before being sent to the conference server via the network. In another embodiment, the fusion model is coupled to the capture device as an integrated system. The discussion herein describes a conference server for purposes of illustration and not by way of limitation.

[0130] The conference server decompresses, decodes, or decrypts the data as needed. The audiovisual data can be fused and analyzed by the AI ​​application using the LSTM model, for example, to identify or infer features in the audiovisual data, such as, but not limited to: audio direction; speaker location in an image; speaker movement; voice signature; facial signature; gestures; and / or objects. In one example, the AI ​​application may require voice recognition or facial recognition. The LSTM model can be trained using data specific to the AI ​​application (which uses sensor data). In one embodiment, more than one model or analysis engine may be used, as described above.

[0131] In one embodiment, speech can be identified and gesture recognition using video data can be performed. The LSTM model can use the identified speech and recognized gestures to provide possible fusion of data and send possible results to the AI ​​application. In one example, gestures combined with voice commands can provide specific control commands to the AI ​​application. In one example, analysis of video data can indicate eye gaze or track eye movement to infer where the user is looking. Eye gaze analysis can generate control commands for AI applications and can be different based on fusion with audio data. In one embodiment, the LSTM model can be trained for a specific AI application and provide control or commands for the application based on fusion data. In another embodiment, the LSTM model can be more general and provide possible relevant data to the AI ​​application, such as audio streams (with speaker ID and location in the environment) and video streams for each speaker to further process and interpret the input. In this example, the AI ​​application can use audio stream and video stream input to derive appropriate commands or perform operations.

[0132] One embodiment utilizes a fisheye camera with a 12MP sensor. Another embodiment may include an infrared (IR) or other depth sensor to provide three-dimensional (3D) or depth information. If there are not enough depth sensors to cover the entire HFOV, the depth information may not be available in 360°. Variants of the capture device may be provided to accommodate various price points acceptable to a wide range of users, or for different applications. For example, the inclusion of a depth sensor or a higher resolution sensor may increase the cost or complexity of the device beyond what is required for the selected AI application.

[0133] Fig.19 1 is a flow chart illustrating a computer-implemented method 1900 for customizing output based on user preferences according to an example embodiment. The operations in method 1900 are performed by a conference server or system (e.g., conference server 135) using the above-described components. Therefore, method 1900 is described by way of example with reference to a conference server. However, it should be understood that at least some of the operations of method 1900 may be deployed on various other hardware configurations, or performed by similar components residing elsewhere in a network environment. Therefore, method 1900 is not intended to be limited to a conference server.

[0134] In operation 1910, a conference server receives audio streams from a plurality of distributed devices. In an example embodiment, the audio streams include speech captured by one or more of the plurality of distributed devices during a conference of two or more users. In some embodiments, the conference is an ad-hoc conference. In these embodiments, the server may perform blind beamforming or continuous speech separation on the received audio streams to separate speech from background noise or different speakers speaking simultaneously into separate audio channels. In some cases, the audio streams are compared to determine that the audio streams represent sounds from the (same) ad-hoc conference. A conference instance is then generated to process the audio streams identified as being from the ad-hoc conference.

[0135] In operation 1920, the identity of a user of one of the distributed devices is identified by the conference server. In one embodiment, the user is identified based on a video signal captured by a camera associated with the conference (e.g., camera 155, camera 1521). The video signal is transmitted to the conference server. The conference server compares the image of the user from the video signal with the stored images of known (e.g., registered) users to determine a match. If the stored image matches the captured image of the user in the video signal, the user is identified. In one embodiment, the image of the user is stored or associated with the user profile of the user.

[0136] In an alternative embodiment, users are identified based on voice signatures. In this embodiment, the voice from the audio stream is parsed or logged and compared to stored voice signatures of known users. If the stored voice signature matches the parsed / logged voice from the audio stream, the user is identified. In one embodiment, the user's voice signature is stored or associated with the user profile of the user.

[0137] In operation 1930, the language preference of the identified user is determined. In some embodiments, the user profile of the identified user is accessed. The user profile includes at least a predetermined preference for the user's language. In some cases, the predetermined preference is established by the user (e.g., explicitly indicated). In other cases, the predetermined preference is determined based on the device configuration of the device associated with the user (e.g., a distributed device such as a cellular phone or a laptop computer). For example, the device can be configured to run in English or Chinese.

[0138] In operation 1940, the conference server generates a transcript as described above. In an example embodiment, speech from an audio stream is converted to text to generate a text-based transcript or a digital transcript. In one embodiment, as described above, a real-time transcript is generated based on a sequence of short words. Late fusion of data can be performed by speech recognition of multiple audio channels processed in parallel to produce phrases. Phrases derived from multiple audio channels are combined in real time or near real time. In one embodiment, approximately two seconds of speech are combined. As a result, audio streams are processed substantially as they are received. Non-overlapping sliding windows of several seconds (such as two seconds) are used to process the audio streams, thereby reducing the latency of transcription generation.

[0139] In operation 1950, the conference server translates the transcript according to the user's language preference. In some embodiments, the conference server obtains the generated transcript from operation 1940 and translates the text in the generated transcript into text in the preferred language. In other embodiments, the conference server obtains the generated transcript from operation 1940 and converts the generated transcript into speech in the preferred language. Further, some embodiments may perform both text translation and speech conversion.

[0140] In operation 1960, the translated transcript is provided to the user's device (e.g., a distributed device). In some embodiments, the device includes the same device used to capture audio from the user. The translated transcript can be provided, for example, as text displayed on a display device (e.g., a screen) of the device, or as voice audio via a speaker device (e.g., a handset, a hearing aid, or a speaker) using text to speech. In some embodiments, a journaled result can also be provided.

[0141] Although Fig.19The method 1900 is described as having a particular order of operations, but alternative embodiments may perform the method 1900 in a different order of operations. For example, identifying the user (operation 1920) and determining the language preference (operation 1930) may occur after or while the transcript is generated (operation 1940) and before the transcript is translated (operation 1950).

[0142] Fig. 20 2 is a block diagram of a computer system 2000 to implement and manage the processing of intelligent conferences via multiple distributed devices, edge devices, and cloud-based devices and to perform methods and algorithms according to example embodiments. Not all components need to be used in various embodiments.

[0143] An example computing device in the form of a computer 2000 may include a processing unit 2002, a memory 2003, a removable storage device 2010, and a non-removable storage device 2012. Although the example computing device is shown and described as a computer 2000, the computing device may take different forms in different embodiments. For example, the distributed device may be a smart phone, a tablet computer, a smart watch, or other computing device that includes information related to Fig. 20 The same or similar elements shown and described. Devices such as smartphones, tablets and smart watches are often collectively referred to as mobile devices, distributed devices or user devices.

[0144] Although various data storage elements are illustrated as being part of the computer 2000, the storage may also or may alternatively include cloud-based storage accessible via a network, such as the Internet or server-based storage or a smart storage device (SSD). It should also be noted that the SSD may include a processor, with the parser running on the processor, thereby allowing the parsed filtered data to be transferred via an I / O channel between the SSD and the main memory.

[0145] The memory 2003 may include volatile memory 2014 and non-volatile memory 2008. The computer 2000 may include, or may have access to, a computing environment that includes, various computer-readable media, such as volatile memory 2014 and non-volatile memory 2008, removable storage 2010, and non-removable storage 2012. Computer storage includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0146] The computer 2000 may include or may access a computing environment including an input interface 2006, an output interface 2004, and a communication interface 2016. The output interface 2004 may include a display device such as a touch screen, which may also be used as an input device. The input interface 2006 may include one or more of the following: a touch screen, a touch pad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the computer 2000 or coupled to the computer 2000 via a wired or wireless data connection, and other input devices. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device, or other public data flow network switches, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular network, Wi-Fi, Bluetooth, or other networks. According to one embodiment, the various components of the computer 2000 are connected to the system bus 2020.

[0147] Computer readable instructions stored on a computer readable medium may be executed by a processing unit 2002 of the computer 2000, such as a program 2018. In some embodiments, the program 2018 includes software to implement one or more methods for implementing a conference application and a conference server, as well as the modules, methods, and algorithms described herein. Hard drives, CD-ROMs, and RAM are some examples of items that include non-transient computer readable media (such as storage devices). The terms "computer readable medium" and "storage devices" do not include carrier waves, as carrier waves are considered transient. Storage devices may also include network storage devices, such as a storage area network (SAN). The computer program 2018 and the workspace manager 2022 may be used to cause the processing unit 2002 to perform one or more methods or algorithms described herein.

[0148] Example:

[0149] 1. A computer-implemented method comprising: receiving audio signals representing speech via multiple audio streams transmitted from corresponding multiple distributed devices; performing continuous speech separation for one or more of the received audio signals having overlapping speech via a neural network model; and providing the separated speech on a fixed number of separate output audio channels.

[0150] 2. A method according to Example 1, wherein continuous speech separation is performed by a neural network model, which is trained using permutation invariant training.

[0151] 3. A method according to Example 2, wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and the locations of distributed devices during a conference between multiple users.

[0152] 4. A method according to any of Examples 1-3, wherein multiple devices capture audio signals during an ad hoc conference.

[0153] 5. The method of any of Examples 1-4, wherein the audio signal is received at a conference server coupled to the distributed device via a network.

[0154] 6. A method according to any of Examples 1-5, and also including generating a transcript based on the separated audio channels.

[0155] 7. The method of example 6, and further comprising including speaker attribution in the generated transcript.

[0156] 8. The method of example 7, further comprising sending the transcript to one or more of the distributed devices.

[0157] 9. A method according to any of Examples 1-8, wherein at least two of the audio streams are provided by an environment capture device having a microphone array at a fixed position.

[0158] 10. A machine-readable storage device having instructions for execution by a processor of a machine, causing the processor to perform operations to execute a method, the operations comprising: receiving an audio signal representing speech via multiple audio streams transmitted from corresponding multiple distributed devices; performing continuous speech separation on one or more audio signals in the received audio signals having overlapping speech via a neural network model; and providing separated speech on a fixed number of separate output audio channels.

[0159] 11. An apparatus according to Example 10, wherein continuous speech separation is performed by a neural network model, and the neural network model is trained using permutation invariant training.

[0160] 12. An apparatus according to example 11, wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and the locations of distributed devices during a conference between multiple users.

[0161] 13. A device according to any of Examples 10-12, wherein the plurality of distributed devices include wireless devices associated with speakers in a conference.

[0162] 14. The device of any of examples 10-13, wherein the audio signal is received at a conference server coupled to the distributed device via a network.

[0163] 15. The apparatus of any of Examples 10-14, further comprising generating a transcript attributed to a speaker based on the separated audio channels.

[0164] 16. The device of example 15 further comprising sending the transcript to one or more of the distributed devices.

[0165] 17. A device according to any of Examples 10-16, wherein at least two of the audio streams are provided by an environment capture device having a microphone array at a fixed position.

[0166] 18. A device comprising a processor and a memory device, the memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations. The operations include: receiving audio signals representing speech via a plurality of audio streams transmitted from a corresponding plurality of distributed devices; performing continuous speech separation for one or more of the received audio signals having overlapping speech via a neural network model; and providing the separated speech on a fixed number of separate output audio channels.

[0167] 19. An apparatus according to Example 18, wherein performing continuous speech separation is performed by a neural network model that is trained using permutation invariant training, and wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and the locations of distributed devices during a conference between multiple users.

[0168] 20. A device according to any of Examples 18-19, wherein the audio signal is received at a conference server coupled to the distributed devices via a network, and wherein the conference server performs an adding operation including: generating a transcript attributed to the speaker based on the separated audio channels, and sending the transcript to one or more of the distributed devices.

[0169] Although several embodiments have been described in detail above, other modifications are possible. For example, the logic flow depicted in the figure does not require the particular order or order of sequence shown to achieve the desired result. From the described flow, other steps can be provided or steps can be eliminated, and other components can be added or removed from the described system. Other embodiments may be within the scope of the appended claims.

Claims

1. A computer-implemented method comprising: receiving an audio signal representing speech via a plurality of audio streams transmitted from a corresponding plurality of distributed devices; detecting overlapping speech during a first time period during the conference; detecting non-overlapping speech during a second time period during the conference; In response to detecting the overlapping speech, performing continuous speech separation on one or more of the received audio signals with the overlapping speech via a neural network model for the first time period, wherein the neural network model includes a local observer including a set of stacked attention layers that map each audio signal into a representation; providing separated said speech on a fixed number of separate output audio channels; and The non-overlapping speech for the second time period is provided on another output audio channel without performing continuous speech separation.

2. The method of claim 1, wherein performing continuous speech separation is performed by the neural network model, which is trained using permutation invariant training.

3. The method of claim 2, wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and locations of distributed devices during a conference between multiple users. The method of claim 1 , wherein the plurality of devices capture the audio signals during an ad hoc conference.

5. The method of claim 1, wherein the audio signal is received at a conference server coupled to the distributed device via a network.

6. The method of claim 1, and further comprising generating a transcript based on the separated audio channels.

7. The method of claim 6, and further comprising including speaker attribution in the generated transcript.

8. The method of claim 7, and further comprising sending the transcript to one or more of the distributed devices.

9. The method of claim 1, wherein at least two of the audio streams are provided by an environment capture device having a microphone array at a fixed location.

10. A machine-readable storage device having instructions for execution by a processor of a machine, causing the processor to perform operations to perform a method, the operations comprising: receiving an audio signal representing speech via a plurality of audio streams transmitted from a corresponding plurality of distributed devices; detecting overlapping speech during a first time period during the conference; detecting non-overlapping speech during a second time period during the conference; In response to detecting the overlapping speech, performing continuous speech separation on one or more of the received audio signals with the overlapping speech via a neural network model for the first time period, wherein the neural network model includes a local observer including a set of stacked attention layers that map each audio signal into a representation; providing separated said speech on a fixed number of separate output audio channels; and The non-overlapping speech for the second time period is provided on another output audio channel without performing continuous speech separation.

11. The apparatus of claim 10, wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training.

12. The device of claim 11, wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and locations of distributed devices during a conference between multiple users.

13. The device of claim 10, wherein the plurality of distributed devices comprises wireless devices associated with speakers in a conference.

14. The device of claim 10, wherein the audio signal is received at a conference server coupled to the distributed device via a network.

15. The apparatus of claim 10, and further comprising generating a transcript attributed to a speaker based on the separated audio channels.

16. The device of claim 15, and further comprising sending the transcript to one or more of the distributed devices.

17. The device of claim 10, wherein at least two of the audio streams are provided by an environment capture device having a microphone array at a fixed location.

18. A device comprising: processor; as well as A memory device coupled to the processor and having stored thereon a program for execution by the processor to perform operations including: receiving an audio signal representing speech via a plurality of audio streams transmitted from a corresponding plurality of distributed devices; detecting overlapping speech during a first time period during the conference; detecting non-overlapping speech during a second time period during the conference; In response to detecting the overlapping speech, performing continuous speech separation on one or more of the received audio signals with the overlapping speech via a neural network model for the first time period, wherein the neural network model includes a local observer including a set of stacked attention layers that map each audio signal into a representation; providing separated said speech on a fixed number of separate output audio channels; and The non-overlapping speech for the second time period is provided on another output audio channel without performing continuous speech separation.

19. The device of claim 18, wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training, and wherein the neural network model is configured to receive a varying number of inputs to support dynamic changes in the number of audio signals and locations of distributed devices during a conference between multiple users.

20. The device of claim 18, wherein the audio signal is received at a conference server coupled to the distributed device via a network, and wherein the conference server performs an adding operation comprising: generating a transcript attributed to the speaker based on the separated audio channels; as well as The transcript is sent to one or more of the distributed devices.

Images