Online speaker separation using local and global clustering

Abstract

In one embodiment, a method includes obtaining at least a portion of an audio stream including voice activity, where at least the portion of the audio stream includes a plurality of segments; generating an embedded vector representing each of the plurality of segments; within each of the local windows, clustering the embedded vectors into one or more clusters to perform speaker recognition, where different clusters correspond to different speakers; presenting at least one first sequence of speaker identifications based on the speaker identification performed for each of the local windows; within each of the global windows, clustering the embedded vectors into one or more clusters to perform speaker recognition, where each of the global windows is longer than each of the local windows; and presenting at least one second order of speaker identifications based on the speaker identification performed for each of the global windows.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning systems. More specifically, the present disclosure relates to online speaker diarization using local and global clustering. Background Art

[0002] Speaker separation refers to the process of identifying or labeling different speakers in an audio stream, and speaker separation is often used to identify "who is speaking when" in a multi-speaker conversation setting. Speaker separation has received a lot of attention recently due to its potential in various applications such as conference conversation analysis and multimedia information retrieval. As a specific example, for conference conversation analysis, speaker separation can potentially be used as a front-end component of automatic speech recognition (ASR), thereby providing improved ASR accuracy and richer analysis depending on the speaker. Summary of the invention

[0003] Technical Solutions

[0004] The present disclosure relates to online speaker separation using local and global clustering.

[0005] In an embodiment of the present disclosure, a method for performing online speaker separation includes: obtaining at least a portion of an audio stream containing speech activity, wherein at least the portion of the audio stream includes multiple segments; generating an embedding vector representing each of the multiple segments; within each of local windows, clustering the embedding vector into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers; presenting at least one first speaker identification sequence based on the speaker identification performed for each of the local windows; within each of global windows, clustering the embedding vector into one or more clusters to perform speaker identification, wherein each of the global windows is longer than each of the local windows; and presenting at least one second speaker identification sequence based on the speaker identification performed for each of the global windows.

[0006] In an embodiment of the present disclosure, an apparatus comprises at least one processing device configured to obtain at least a portion of an audio stream containing voice activity, wherein at least the portion of the audio stream comprises a plurality of segments; generate an embedding vector representing each of the plurality of segments; within each of local windows, cluster the embedding vector into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers; based on the speaker identification performed for each of the local windows, initiate the presentation of at least one first speaker identification sequence; within each of global windows, cluster the embedding vector into one or more clusters to perform speaker identification, wherein each of the global windows is longer than each of the local windows; and initiate the presentation of at least one second speaker identification sequence based on the speaker identification performed for each of the global windows.

[0007] In an embodiment of the present disclosure, a computer-readable medium includes instructions that, when executed, cause at least one processor to obtain at least a portion of an audio stream containing voice activity, wherein at least the portion of the audio stream includes a plurality of segments. The non-transitory computer-readable medium also includes instructions that, when executed by at least one processor, are used to generate an embedding vector representing each of the plurality of segments within each of the local windows. The non-transitory computer-readable medium also includes instructions that, when executed by at least one processor, are used to cluster the embedding vectors into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers. The non-transitory computer-readable medium also includes instructions that, when executed by at least one processor, are used to initiate presentation of at least one first speaker identification sequence based on the speaker identification performed for each of the local windows, and within each of the global windows, cluster the embedding vectors into one or more clusters to perform speaker identification, wherein each of the global windows is longer than each of the local windows. The non-transitory computer-readable medium also includes instructions that, when executed by at least one processor, are used to initiate presentation of at least one second speaker identification sequence based on the speaker identification performed for each of the global windows.

[0008] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0010] FIG1 illustrates an example network configuration including electronic devices according to the present disclosure;

[0011] FIG2 illustrates an example architecture for online speaker separation using local clustering and global clustering according to the present disclosure;

[0012] FIG3 illustrates an example of local clustering in the architecture of FIG2 according to the present disclosure;

[0013] FIG4 illustrates an example of global clustering in the architecture of FIG2 according to the present disclosure;

[0014] FIG5 illustrates an example process for assigning speaker identifications to compensate for cluster permutations according to the present disclosure;

[0015] FIG6 illustrates example results obtained using online speaker separation using local clustering and global clustering according to the present disclosure; and

[0016] FIG. 7 illustrates an example method for online speaker separation using local clustering and global clustering according to the present disclosure. DETAILED DESCRIPTION

[0017] It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include both direct and indirect communications. The terms "include," and "comprise," and their derivatives, mean including, but not limited to, this. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means include, be included within, be interconnected with, include, be contained within, be connected to or connected with, be coupled to or coupled with, be communicable with, collaborate with, be intertwined, be juxtaposed, be close to, be bound to or bound with, have, have the nature of, have a relationship with, etc.

[0018] In addition, the various functions and modules described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, modules, objects, classes, instances, related data or a portion thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical disks or erasable memory devices.

[0019] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component and vice versa.

[0020] It should be understood that when an element (such as a first element) is referred to as being (operably or communicatively) “coupled” or “connected” to or “connected to” another element (such as a second element), it may be coupled or connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being “directly coupled” or “directly coupled to” another element (such as a second element) or “directly connected to” another element (such as a second element), there are no other elements (such as a third element) between the element and the other element.

[0021] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." Instead, the phrase "configured to" may mean that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a dedicated processor (such as an embedded processor) for performing operations.

[0022] The terms and phrases used herein are only used to describe embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly states otherwise. All terms and phrases used herein, including technical and scientific terms and phrases, have the same meanings as those commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in commonly used dictionaries) should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0023] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as Samsung HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as Samsung GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a portable camera, or an electronic photo frame. Other examples of electronic devices include at least one of the following: various medical devices (such as various portable medical measuring devices (such as blood sugar measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle head units, industrial or household robots, automated teller machines (ATMs), point of sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electric or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a portion of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to an embodiment of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to an embodiment of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include new electronic devices depending on the development of technology.

[0024] In the following description, an electronic device is described with reference to the accompanying drawings according to an embodiment of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0025] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0026] Nothing in the description of this application should be interpreted as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of the patented subject matter is limited only by the claims. Applicants understand that any other terms, including but not limited to "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor" or "controller" are used in the claims to refer to structures known to those skilled in the relevant art.

[0027] The embodiments of the present disclosure are described with reference to the accompanying drawings, which are discussed below in Figures 1 to 7. However, it should be understood that the present disclosure is not limited to the embodiments, and all changes and / or equivalents or substitutions thereof also belong to the scope of the present disclosure. Throughout the specification and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0028] As mentioned above, speaker separation refers to the process of identifying or labeling different speakers in an audio stream, and speaker separation is often used to identify "who is speaking when" in a multi-speaker conversation setting. Speaker separation has received a lot of attention recently due to its potential in various applications such as conference conversation analysis and multimedia information retrieval. As a specific example, for conference conversation analysis, speaker separation can potentially be used as a front-end component of automatic speech recognition (ASR), thereby providing improved ASR accuracy and richer analysis depending on the speaker.

[0029] Various methods have been developed for performing speaker separation in an offline or non-real-time manner. These methods typically process a representation of the audio data once the entire audio recording is available, which means that these methods cannot be used to process the audio data in a real-time or streaming manner when the audio data becomes available. Although there are some methods for performing speaker separation in an online or real-time manner, these methods may have various disadvantages. For example, online speaker separation methods typically have significantly worse accuracy than offline speaker separation methods. In addition, online speaker separation methods typically use machine learning models trained in a supervised manner, so these machine learning models typically lack flexibility and modularity. In addition to this, it is difficult to generalize these machine learning models to arbitrary domains and environments, and it is difficult to maintain these machine learning models after deployment.

[0030] The present disclosure provides various techniques for online speaker separation using local and global clustering. As described in more detail below, at least a portion of an audio stream containing voice activity is obtained (such as by performing voice activity detection), and at least the portion of the audio stream is divided into segments (such as segments of one second to one and a half seconds in length). For each segment, an embedding vector representing the segment is generated, such as by identifying features of the segment. In response to generating a first set of embedding vectors for a first set of segments received during a first global window, the first set of embedding vectors can be clustered into one or more clusters (where different clusters correspond to different speakers) to perform speaker identification, and a first speaker identification sequence based on speaker identification performed for the first global window is generated and presented. In response to generating a second set of embedding vectors for a second set of segments received during a second global window, the second set of embedding vectors can be clustered to perform speaker identification, and a second speaker identification sequence based on speaker identification performed for the second global window is generated and presented. The second global window can include the first global window and be longer than the first global window. Local clustering can also occur within each global window. For example, local clustering can occur during multiple partially overlapping local windows within each global window, and each local clustering can involve clustering a subset of the embedding vectors. Additionally, timestamps can be used to support consistent recognition of the same speaker across different local and global clustering operations.

[0031] In this way, the techniques for online speaker separation described in the present disclosure can support multi-level clustering. Local clustering can occur in order to meet the desired delay requirements, because when processing audio data over a shorter time span, monotonically increasing block-based spectral clustering or other local clustering techniques can be performed faster. Global clustering can occur in order to meet the desired accuracy requirements, because when processing audio data over a longer time span, global-wise spectral clustering or other global clustering techniques can be more accurate. Therefore, the techniques for online speaker separation described in the present disclosure can meet various accuracy and streaming capability requirements required or desired in various applications. In general, the techniques for online speaker separation described in the present disclosure can be used to provide speaker separation results in real time, and these results can be much more accurate than other online techniques, and may be close to or as accurate as offline techniques.

[0032] The online speaker separation technology described in this disclosure can be used in a large number of possible applications. For example, an online transcription application may involve converting audio data into a text-based transcription in real time. A conference conversation analysis application may involve processing audio data associated with a conference or other conversation in order to analyze the conversation. A multimedia information retrieval application may involve processing multimedia content (such as based on conversations contained in the multimedia content) and retrieving multimedia content that satisfies a user query. All of these example applications may rely on, use, or be supported by online speaker separation technology in order to correctly perform the functions described therein. Of course, the online speaker separation functionality described in this document may be used in any other suitable manner.

[0033] FIG1 illustrates an example network configuration 100 including electronic devices according to the present disclosure. The embodiment of the network configuration 100 shown in FIG1 is for illustration only. In other aspects, embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0034] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In an embodiment, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting the components 120-180 to each other and for transmitting communications (such as control messages and / or data) between the components.

[0035] The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing communication-related operations or data processing. In an embodiment of the present disclosure, the processor 120 may be a graphics processor unit (GPU). As described below, the processor 120 may be used to perform online speaker separation.

[0036] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0037] The kernel 141 may control or manage system resources (such as bus 110, processor 120, or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145, or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145, or application 147 to access various components of the electronic device 101 to control or manage system resources. Application 147 includes one or more applications related to online speaker separation. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 may be used as a repeater to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning a priority to use the system resources (such as bus 110, processor 120, or memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

[0038] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0039] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body part of the user.

[0040] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0041] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more cameras or other imaging sensors, which can be used to capture images of scenes. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located within the electronic device 101.

[0042] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0043] Wireless communication can use, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM) as a cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). Network 162 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0044] The first external electronic device 102 and the second external electronic device 104 and the server 106 may each be a device of the same or different type as the electronic device 101. According to some embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to some embodiments of the present disclosure, all or some of the operations performed on the electronic device 101 may be performed on another one or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to some embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform a certain function or service, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the function or service alone or in addition. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested function or additional function and transmitting the execution result to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or additionally. To this end, for example, cloud computing, distributed computing or client-server computing technology may be used. Although FIG. 1 illustrates that the electronic device 101 includes the communication interface 170 to communicate with the external electronic device 104 or the server 106 via the network 162 , according to an embodiment of the present disclosure, the electronic device 101 may independently operate without a separate communication function.

[0045] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or a processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may be used to perform online speaker separation.

[0046] Although FIG. 1 shows an example of a network configuration 100 including an electronic device 101, various changes may be made to FIG. 1. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a variety of configurations, and FIG. 1 does not limit the scope of the present disclosure to any particular configuration. In addition, although FIG. 1 shows an operating environment in which various features disclosed in this patent document may be used, these features may be used in any other suitable system.

[0047] 2 shows an example architecture 200 for online speaker separation using local clustering and global clustering according to the present disclosure. For ease of explanation, the architecture 200 is described as being used by the electronic device 101, the server 106, or other devices in the network configuration 100 of FIG. 1. However, the architecture 200 may be used by any other suitable device and in any other suitable system.

[0048] As shown in FIG. 2 , the architecture 200 generally operates to receive and process input audio data 202. The audio data 202 generally includes audio data of multiple people speaking. Generally, the audio data 202 may additionally include other audio data, such as background noise or other non-speech data. The audio data 202 may be obtained from any suitable source. In some cases, the audio data 202 may be captured by the electronic device 101, such as when the audio data 202 is captured by at least one microphone (at least one sensor 180) of the electronic device 101. Note that the audio data 202 may or may not be part of a larger multimedia content collection, such as when the audio data 202 is associated with corresponding video data.

[0049] The audio data 202 is received at and processed by the voice activity detection module 204, which analyzes the audio data 202 to identify portions of the audio data 202 that contain voice activity of one or more speakers. For example, the voice activity detection module 204 can be used to automatically detect frames of the audio data 202 that contain voice activity and filter out frames of the audio data 202 that contain non-speech signals. Frames containing voice activity can also be divided into shorter audio data segments. Each segment of the audio data 202 can have any suitable duration, such as between one second and one and a half seconds (although other durations are possible). The voice activity detection module 204 can use any suitable technique to identify segments or other portions of the audio data 202 that contain voice activity.

[0050] The segments of the audio data 202 containing speech activity are provided to the speaker embedding extraction module 206, which generates an embedding vector representing the segment of the audio data 202. For example, the embedding vector may represent a feature of the audio data 202, which is extracted or otherwise generated by the speaker embedding extraction module 206 and placed into an array to form the embedding vector. The feature may represent a characteristic or other aspect of the audio data 202 that bears on the identification of the speaker in the audio data 202. The speaker embedding extraction module 206 may use any suitable technique to extract features of the audio data or otherwise generate an embedding vector for the audio data. In an embodiment of the present disclosure, the speaker embedding extraction module 206 represents a machine learning model, such as a convolutional neural network (CNN), a deep learning network (DNN), or other machine learning model. Using suitable training data (such as training audio data with known speakers), the machine learning model can be trained to identify which features of the training audio data are relevant to speaker identification, and the trained machine learning model can be used to extract those relevant features from the audio data 202 for generating an embedding vector.

[0051] The embedding vectors are provided to a local speaker clustering module 208 and a global speaker clustering module 210. Each speaker clustering module 208 and 210 is configured to cluster the embedding vectors into different clusters, where the different clusters correspond to different speakers. For example, a first cluster may correspond to a first speaker, and a second cluster may correspond to a second speaker. For example, each embedding vector may represent a feature of a segment of the audio data 202 within a feature space, and each speaker clustering module 208 and 210 may identify clusters or groups of embedding vectors within the feature space. As a specific example, each speaker clustering module 208 and 210 may use cosine similarity or other measures of similarity to identify groups of embedding vectors that are more related to each other and less related to other groups of embedding vectors. When the features forming the embedding vectors represent spectral features of the audio data 202, the local speaker clustering module 208 may be said to perform locally-wise spectral clustering, and the global speaker clustering module 210 may be said to perform globally-wise spectral clustering.

[0052] Speaker clustering modules 208 and 210 differ in which embedding vectors are clustered by clustering modules 208 and 210. For example, local speaker clustering module 208 may perform speaker identification by clustering embedding vectors generated within smaller time windows. In some cases, the duration of these smaller time windows (referred to as local windows) may be five minutes, but other durations may be used. Global speaker clustering module 210 may perform speaker identification by clustering embedding vectors generated within larger time windows, where the larger time windows include the smaller time windows. In some cases, these larger time windows (referred to as global windows) may have a minimum duration of twenty minutes and may extend in duration as more and more audio data 202 is received, but other durations may be used. For example, each of the global windows is longer than each of the local windows.

[0053] The local speaker clustering module 208 processes the embedding vectors corresponding to the segments of the audio data 202 received in the smaller local time window. In an embodiment of the present disclosure, the local speaker clustering module 208 performs monotonically increasing block-based spectral clustering or other local clustering. As described in more detail below, the local speaker clustering module 208 can perform clustering on a first subset of available embedding vectors within a first local window, perform clustering on a second (larger) subset of available embedding vectors within the first local window, and continue until all embedding vectors within the first local window are clustered. The time window can slide or otherwise move forward to define a second local window, which can overlap with the first local window. The local speaker clustering module 208 can perform clustering on a first subset of available embedding vectors within a second local window, perform clustering on a second (larger) subset of available embedding vectors within the second local window, and continue until all embedding vectors within the second local window are clustered. In some cases, the time window used by the local speaker clustering module 208 may have a fixed length and may slide forward as more audio data 202 is received and converted into embedding vectors over time. An example of this type of approach for local clustering is shown in FIG3 , which is described in more detail below.

[0054] The global speaker clustering module 210 processes embedding vectors corresponding to segments of the audio data 202 received in a larger global time window. In an embodiment of the present disclosure, the global speaker clustering module 210 performs global spectrum clustering or other global clustering. As described in more detail below, the global speaker clustering module 210 can perform clustering in a larger global time window, and the time window can be extended in duration. In some cases, the time window used by the global speaker clustering module 210 may initially have a smaller duration, such as a duration containing two (possibly overlapping) local windows, and the time window used by the global speaker clustering module 210 may be extended to include more than two local windows over time. As a result, the global speaker clustering module 210 can process embedding vectors corresponding to the audio data 202 received in the first global window, process embedding vectors corresponding to the audio data 202 received in the second global window including the first global window, and so on. An example of this type of method for global clustering is shown in FIG. 4, which is described in more detail below.

[0055] In this example, the local speaker clustering module 208 is used to generate partial separation results 212, and the global speaker clustering module 210 is used to generate final separation results 214. The partial separation results 212 represent the separation results generated by the local speaker clustering module 208 using clustering in a local window. These results 212 can usually be generated relatively quickly and are useful for stream-ability or other purposes involving fast separation. However, the results 212 may not be as accurate as the final separation results 214 because the final separation results 214 represent the separation results generated by the global speaker clustering module 210 using clustering in a larger global window. As a result, in some cases, the partial separation results 212 can be output for use (e.g., for immediate display) and can be overwritten by the final separation results 214 later (because the final separation results 214 can replace some of the results included in the partial separation results 212). In this way, the architecture 200 can use local clustering to meet the desired latency requirements and use global clustering to meet the desired accuracy requirements.

[0056] In an embodiment of the present disclosure, each of speaker clustering modules 208 and 210 represents a trained machine learning model, such as a CNN, DNN, or other machine learning model. Using suitable training data (such as training embedding vectors associated with known speakers), the machine learning model can be trained to cluster the training embedding vectors and identify clusters of embedding vectors associated with known speakers. The trained machine learning model can be used to cluster the embedding vectors generated using audio data 202 in order to identify different speakers within audio data 202.

[0057] It should be noted that the modules shown in FIG. 2 or described with respect to FIG. 2 can be implemented in the electronic device 101, the server 106, or other devices in any suitable manner. For example, in an embodiment of the present disclosure, one or more software applications or other software instructions executed by the processor 120 of the electronic device 101, the server 106, and / or other devices can be used to implement or support at least some of the modules shown in FIG. 2 or described with respect to FIG. 2. On the other hand, in an embodiment of the present disclosure, dedicated hardware components can be used to implement or support at least some of the modules shown in FIG. 2 or described with respect to FIG. 2. In general, any suitable hardware or any suitable combination of hardware and software / firmware instructions can be used to perform the functions shown in FIG. 2 or described with respect to FIG. 2. In addition, the modules shown in FIG. 2 or described with respect to FIG. 2 can be performed by a single device or by multiple devices. In some cases, for example, the modules shown in FIG. 2 or described with respect to FIG. 2 can be executed by the electronic device 101, and the separation results 212-214 can be displayed or otherwise used by the electronic device 101 in any suitable manner. In other cases, electronic device 101 may obtain audio data 202 and provide it to server 106 for processing, and server 106 may provide separation results 212-214 to electronic device 101 or other suitable destinations, or may use separation results 212-214 in any suitable manner. In general, the present disclosure is not limited to any particular physical implementation of architecture 200.

[0058] Although Figure 2 shows an example of an architecture 200 for online speaker separation using local clustering and global clustering, various changes may be made to Figure 2. For example, the various components shown in Figure 2 may be added, omitted, combined, further subdivided, duplicated, or placed in any other suitable configuration according to specific needs.

[0059] FIG3 shows an example of local clustering 300 in the architecture 200 of FIG2 according to the present disclosure. Specifically, FIG3 shows an example of how the local speaker clustering module 208 can cluster and process the embedding vectors generated using the audio data 202. For ease of explanation, the local clustering 300 is described as being performed by the electronic device 101, the server 106, or other devices in the network configuration 100 of FIG1. ​​However, the local clustering 300 can be performed by any other suitable device and in any other suitable system.

[0060] As shown in FIG3, local clustering 300 involves processing of audio data 302, which may be received and processed in real time or in any other suitable manner. Audio data 302 may represent part or all of audio data 202, such as when audio data 302 represents audio data 202 in which voice activity is detected by voice activity detection module 204. Here, audio data 302 is divided into segments 304, wherein each segment 304 contains some form of voice activity detected by one or more speakers. Each segment 304 of audio data 302 may have any suitable duration, such as a duration of about one second to about one and a half seconds (although other durations are possible). Each segment 304 of audio data 302 is converted into a corresponding embedding vector 306, such as a feature vector, by speaker embedding extraction module 206.

[0061] The local speaker clustering module 208 defines a first local window 308 that represents a time period associated with a first smaller collection of segments 304. In this example, the local speaker clustering module 208 performs monotonically increasing block-based spectral clustering or other local clustering of the embedding vectors 306 within the first local window 308. Thus, for example, the local speaker clustering module 208 may perform local clustering on a first subset 310 of the embedding vectors 306 within the first local window 308. Using the first subset 310, the local speaker clustering module 208 clusters the embedding vectors 306 and attempts to identify different speakers associated with different clusters of the embedding vectors 306. When one or more additional embedding vectors 306 become available, the local speaker clustering module 208 may perform local clustering on a second subset 312 of the embedding vectors 306 within the first local window 308, wherein the second subset 312 includes the embedding vectors 306 of the first subset 310 and at least one additional embedding vector 306. Using the second subset 312, the local speaker clustering module 208 again clusters the embedding vectors 306 and attempts to identify different speakers associated with different clusters of the embedding vectors 306. When one or more additional embedding vectors 306 become available, the local speaker clustering module 208 may perform local clustering on a third subset 314 of the embedding vectors 306 within the first local window 308, wherein the third subset 314 includes the embedding vectors 306 of the first subset 310 and the second subset 312 and at least one additional embedding vector 306. Using the third subset 314, the local speaker clustering module 208 again clusters the embedding vectors 306 and attempts to identify different speakers associated with different clusters of the embedding vectors 306. This process may continue until all embedding vectors 306 for all segments 304 of the audio data 302 within the first local window 308 are processed. At this point, processing of the embedding vectors 306 for the first local window is complete. For example, each of the second subset 312 and the third subset 314 may be referred to as a subsequent subset of the first subset 310. In this case, the first subset 310 may be referred to as a previous subset of each of the second subset 312 and the third subset 314.

[0062] The time window used by the local speaker clustering module 208 is shifted to define a second local window 316, and the same type of processing shown for the first local window 308 can be performed in the second local window 316 (but with a different set of embedding vectors 306 for a different set of segments 304). All segments 304 of the audio data 302 within the second local window 316 can be processed in this manner, and the processing of the embedding vectors 306 for the second local window 316 is now complete. The time window used by the local speaker clustering module 208 is shifted again to define a third local window 318, and the same type of processing shown for the first local window 308 can be performed in the third local window 318 (with a different set of embedding vectors 306 for a different set of segments 304). This process can continue for the duration of the audio data 302.

[0063] The results of the clustering in each local window 308, 316, 318 can be used to identify a list or sequence of speaker identifications in each local window 308, 316, 318. Thus, for example, the architecture 200 can identify and output a sequence of speaker identifications in which different speakers are identified as saying different things within each local window 308, 316, 318. In some cases, the speaker identification sequence can be generated and output after processing a first subset 310 of the embedding vectors 306 within the first local window 308. Since speaker separation generally becomes more accurate as more data is processed, an updated speaker identification sequence (which may or may not match the original speaker identification sequence) can be generated and output after processing a second subset 312 of the embedding vectors 306 within the first local window 308. Similarly, after processing a third subset 314 of the embedding vectors 306 within the first local window 308, another updated speaker identification sequence (which may or may not match the previous speaker identification sequence) can be generated and output. In embodiments of the present disclosure, this allows results to be presented in real time or near real time and then updated if necessary.

[0064] Note that the local windows 308, 316, 318 can each have any suitable duration, and the time windows used by the local speaker clustering module 208 to define the local windows 308, 316, 318 can be shifted by any suitable amount to define adjacent local windows. For example, in some cases, each local window 308, 316, 318 can have a duration of five minutes, and the time windows used to define the local windows can be shifted by three minutes. In this case, the first time window 308 can span the time period from t to t+5, the second time window 316 can span the time period from t+3 to t+8, and the third time window 318 can span the time period from t+6 to t+11. Thus, two adjacent local windows can overlap for a period of two minutes. However, other durations, shift amounts, and overlap amounts can be used herein.

[0065] FIG4 shows an example of global clustering 400 in the architecture 200 of FIG2 according to the present disclosure. Specifically, FIG4 shows an example of how the global speaker clustering module 210 can cluster and process the embedding vectors generated using the audio data 202. For ease of explanation, the global clustering 400 is described as being performed by the electronic device 101, the server 106, or other devices in the network configuration 100 of FIG1. ​​However, the global clustering 400 may be performed by any other suitable device and in any other suitable system.

[0066] 4 , local windows 308, 316, 318 are shown for use by the local speaker clustering module 208. Different local windows 308, 316, 318 may be associated with different subsets of the segments 304 of the audio data 302, and the local windows 308, 316, 318 (and thus their associated subsets of the segments 304) may overlap with each other.

[0067] The global speaker clustering module 210 defines a first global window 402 that represents a time period associated with the first larger group of segments 304. In some cases, the first global window 402 may include more than one local window. In this particular example, the first global window 402 includes two of the local windows 308, 316. The global speaker clustering module 210 may perform global spectral clustering or other global clustering of the embedding vectors 306 within the first global window 402. Thus, for example, the global speaker clustering module 210 may perform global clustering on a first set 404 of embedding vectors 306, which may be the same embedding vectors 306 processed by the local speaker clustering module 208 during the local windows 308, 316. Using the first set 404 of embedding vectors 306, the global speaker clustering module 210 clusters the embedding vectors 306 and attempts to identify different speakers associated with different clusters of the embedding vectors 306.

[0068] The global speaker clustering module 210 also defines a second global window 406 that is longer than the first global window 402 and includes the first global window 402 (and is therefore associated with a second, even larger set of segments 304). The global speaker clustering module 210 may perform global spectral clustering or other global clustering of the embedding vectors 306 within the second global window 406. Thus, for example, the global speaker clustering module 210 may perform global clustering on a second set 408 of embedding vectors 306, which may be the same embedding vectors 306 processed by the local speaker clustering module 208 during the local windows 308, 316, 318 (and which include the embedding vectors 306 of the first set 404). Using the second set 408 of embedding vectors 306, the global speaker clustering module 210 clusters the embedding vectors 306 and attempts to identify different speakers associated with different clusters of embedding vectors 306. This process may continue for the duration of the audio data 302.

[0069] For each global window 402, 406, the global speaker clustering module 210 can cluster the embedding vectors 306 and attempt to identify different speakers associated with different clusters of the embedding vectors 306. The results of the clustering in each global window 402, 406 can be used to identify a list or sequence of speaker identifications in each global window 402, 406. Thus, for example, the architecture 200 can identify and output a sequence of speaker identifications in which different speakers are identified as saying different things within each global window 402, 406. In some cases, the speaker identification sequence can be generated and output after processing the embedding vectors 306 within the first global window 402. Since speaker separation generally becomes more accurate as more data is processed, an updated speaker identification sequence (which may or may not match the original speaker identification sequence) can be generated and output after processing the embedding vectors 306 within the second global window 406.

[0070] Here, each global window 402, 406 can have any suitable duration. For example, in some cases, the first global window 402 can have a duration of twenty minutes. As more and more audio data 302 are received, the duration of the global window can be gradually extended so that the global window contains the period from the beginning of the first global window 402 to the current time. In some cases, this can occur in a defined step, such as when the second global window 406 is five minutes, ten minutes, fifteen minutes, or twenty minutes longer than the first global window 402 and each subsequent global window is five minutes, ten minutes, fifteen minutes, or twenty minutes longer than the previous global window. The final global window can span the entire length of the audio data 302.

[0071] Note that portions of the processing during local clustering 300 may occur sequentially or in parallel. For example, clustering of embedding vectors 306 in one local window may be completed before clustering of embedding vectors 306 in the next local window begins, or clustering of embedding vectors 306 in multiple local windows may overlap with each other. Similarly, portions of the processing during global clustering 400 may occur sequentially or in parallel. For example, clustering of embedding vectors 306 in one global window may be completed before clustering of embedding vectors 306 in the next global window begins, or clustering of embedding vectors 306 in multiple global windows may overlap with each other. In addition, portions of the processing during local clustering 300 and global clustering 400 may occur sequentially or in parallel. For example, clustering of embedding vectors in one or more local windows may be completed before clustering of embedding vectors in a global window begins, or clustering of embedding vectors 306 in local and global windows may overlap. In some cases, performing local and global clustering in parallel can help speed up the generation of the final separation result 214.

[0072] Although FIG. 3 and FIG. 4 illustrate examples of local clustering and global clustering in the architecture 200 of FIG. 2 , various changes may be made to FIG. 3 and FIG. 4 . For example, the number of embedding vectors 306 in each subset processed within a local window may vary, and the number of local windows and the length of each local window may vary. Similarly, the number of embedding vectors 306 in each set processed within a global window may vary, and the number of global windows and the length of each global window may vary.

[0073] FIG5 shows an example process 500 for assigning speaker identifications to compensate for cluster permutations according to the present disclosure. For ease of explanation, process 500 is described as being used by electronic device 101, server 106, or other device in network configuration 100 of FIG1. ​​However, process 500 may be used by any other suitable device and in any other suitable system.

[0074] One problem that may arise from using a clustering algorithm (either local or global) is the inconsistent use of identifiers for identified clusters. For example, a clustering algorithm that can potentially cluster embedding vectors into k clusters may use k different cluster identifiers to represent the clusters, but the cluster identifiers may be assigned inconsistently during different clustering operations. An example of this is shown in FIG. 5 , where a first result 502 shows speaker identifiers for a first clustering operation, and a second result 504 shows speaker identifiers for a second clustering operation that includes the same embedding vectors as the first clustering operation. As can be seen here, speaker identifier “1” in the first result 502 has been replaced by speaker identifier “3” in the second result 504, speaker identifier “2” in the first result 502 has been replaced by speaker identifier “1” in the second result 504, and speaker identifier “3” in the first result 502 has been replaced by speaker identifier “2” in the second result 504. The clusters involving the embedding vectors for these three speakers may be the same in the two clustering operations, but the identifiers assigned to the speakers are different.

[0075] To help compensate for this, entries in the previous sequence of speaker identifications may be matched with a portion of entries in the subsequent sequence of speaker identifications. In some cases, this may be done based on timestamps associated with the segments 304 of the audio data 302 used to generate the embedding vectors 306. Once a portion of an entry in the subsequent sequence of speaker identifications matches an entry in the previous sequence of speaker identifications, the matching entry in the subsequent sequence of speaker identifications is replaced by the corresponding entry in the previous sequence of speaker identifications. In FIG. 5 , this involves determining that the first four entries in the second result 504 correspond to the four entries in the first result 502, which may involve matching the timestamps of the entries in the first result 502 and the second result 504. Once matched, the four entries in the second result 504 are replaced by the corresponding entries in the first result 502, resulting in the generation of a modified second result 506. Although not shown here, other entries of the second result 504 (such as additional entries added to the second result 504 and not included in the first result 502) may be updated to ensure that consistent identifiers are used to represent the same speaker in different clustering operations. This allows consistent speaker identifiers to be used in all local and global clustering operations performed in architecture 200. For example, first result 502 may correspond to a first speaker identification sequence for a local clustering operation, and second result 504 may correspond to a second speaker identification sequence for a local clustering operation. For example, first result 502 may correspond to a first speaker identification sequence for a global clustering operation, and second result 504 may correspond to a second speaker identification sequence for a global clustering operation. Entries in the second speaker identification sequence may replace entries in the first speaker identification sequence.

[0076] Although FIG. 5 illustrates one example of a process 500 for assigning speaker identifiers to compensate for cluster permutations, various changes may be made to FIG. 5 . For example, the specific speaker identifiers shown here are for illustration only. Furthermore, although process 500 represents one approach for resolving inconsistent speaker identifiers across different clustering operations, other approaches may also be used. For example, in a global clustering operation, speaker identifiers may be sequentially assigned to clusters of embedding vectors based on the order in which the speakers first appeared in the audio data 202. Because each subsequent global clustering operation may include the same embedding vector 306 used in a previous global clustering operation, this approach may help achieve consistent speaker identifiers across global clustering operations.

[0077] Another method that can be used for local and global clustering operations involves comparing the cluster vectors for each speaker after performing a local or global clustering operation with the cluster vectors for each speaker generated during at least one previous local or global clustering operation. Cosine similarity values ​​or other similarity values ​​can be used to determine whether cluster vectors generated during different local or global clustering operations are associated with the same speaker. As a specific example thereof, an average or other speaker vector can be generated for each speaker based on the clustering of the embedding vectors 306 for that speaker for the completed local or global clustering operation. During a subsequent local or global clustering operation, a cosine similarity value or other similarity value can be determined between the centroid of the cluster of embedding vectors 306 generated during the subsequent local or global clustering operation and the speaker vectors from at least one previous local or global clustering operation. If the similarity value is greater than a specified threshold (such as 0.65), a cluster identifier for a subsequent local or global clustering operation can be assigned to match the same identifier as the identifier used in the previous local or global clustering operation.

[0078] 6 shows an example result 600 obtained by online speaker separation using local clustering and global clustering according to the present disclosure. The result 600 can be obtained, for example, by the electronic device 101, the server 106, or other devices implementing the architecture 200 shown in FIG2. However, the same type of result can be obtained by any other suitable device and in any other suitable system.

[0079] As shown in FIG6 , input audio data 602 is received and processed to produce a transcription 604. The audio data 602 includes speech containing words spoken by multiple people, and the transcription 604 includes the same words, but breaks down who said what during the conversation. In this example, the transcription 604 includes multiple entries, each of which includes a portion 606 of what was said from the audio data 602. Each entry also includes a timestamp 608, which in some cases can identify when the associated portion 606 of the spoken content begins in the audio data 602. Each entry also includes a speaker tag 610, which can be generated by the architecture 200. Each speaker tag 610 provides an identification of the speaker who is determined to be speaking the associated portion 606 of what is said in the audio data 602.

[0080] In this example, speaker labels 610 are given generic labels, namely "Speaker 1", "Speaker 2", and "Speaker 3". However, other labels may be used, such as the actual names of the speakers. The names or other labels 610 used to identify the speakers may be determined in any suitable manner. In some cases, for example, at least one user may provide an identification of each of the one or more speakers at one or more locations within the transcription 604, and the labels of each speaker may be updated within the transcription 604 (possibly throughout the transcription). As another example, natural language processing, automatic speech recognition, or other techniques may be used to identify the labels of speakers based on the content of the audio data 602 itself, such as when one speaker directs a question to a named individual and another speaker (possibly the named individual) answers the question. As yet another example, speaker verification techniques based on speaker vector similarities (such as those based on cosine similarity values) may be used to assign actual speaker names or other labels. Here, if a speaker vector generated from a speaker's previous utterances is available (such as from previous speaker enrollments where the speaker said certain words or phrases, such as "Hi BIXBY" during voice assistant enrollment), speaker similarity can be determined, such as by determining the similarity between the centroid of the cluster being processed and the clusters of the already enrolled speaker vectors. In general, any suitable technique can be used to identify generic or customized labels for speakers captured in the audio data 602.

[0081] The transcription 604, speaker tags 610, or other information here can be further processed for use in any desired manner. For example, the transcription 604 can be displayed (possibly in real time) for use in a streaming service, a closed captioning system, or other system. The transcription 604 can be processed to provide a conversation analysis based on who said what during the conversation. The transcription 604 and the audio data 202 (and possibly other data, such as video data) can be stored, and the transcription 604 can then be searched based on a user query to determine whether the audio data 202 (and possibly other data) should be provided in response to the query. In general, the transcription 604, speaker tags 610, or other information here can be used in any suitable manner.

[0082] Although FIG6 shows one example of a result 600 obtained using online speaker separation using local clustering and global clustering, various changes may be made to FIG6. For example, the actual audio content of the audio data may vary widely, and FIG6 does not limit the scope of the present disclosure to any particular type of audio content. In general, online speaker separation may be used with any suitable audio data containing speech activity, and the results of online speaker separation may be used in any number of ways.

[0083] 7 shows an example method 700 for online speaker separation using local clustering and global clustering according to the present disclosure. For ease of explanation, the method 700 is described as being performed by the electronic device 101, the server 106, or other devices in the network configuration 100 of FIG. 1. However, the method 700 may be performed by any other suitable device and in any other suitable system.

[0084] As shown in FIG. 7 , audio data is obtained at step 702, and segments of the audio data containing voice activity are identified at step 704. This may include, for example, the processor 120 of the electronic device 101 or the server 106 obtaining the audio data 202 from any suitable source, such as from one or more microphones of the electronic device 101 or other device. This may also include the processor 120 of the electronic device 101 or the server 106 executing the voice activity detection module 204 to identify the segments 304 of the audio data containing voice activity. An embedding vector of the segment of the audio data is generated at step 706. This may include, for example, the processor 120 of the electronic device 101 or the server 106 executing the speaker embedding extraction module 206 to generate a feature vector or other embedding vector 306 representing the segment 304.

[0085] In step 708, the embedding vectors in the plurality of local windows are clustered to perform speaker identification, and in step 710, one or more speaker identification sequences are presented based on the speaker identification performed in one or more local windows. This may include, for example, the processor 120 of the electronic device 101 or the server 106 executing the local speaker clustering module 208 to cluster the embedding vectors 306 generated for the segments 304 received within the respective local windows 308, 316, 318. This may also include the processor 120 or the server 106 of the electronic device 101 generating and outputting the partial separation result 212 based on the local clustering operation. The local windows here may include windows that partially overlap each other.

[0086] In step 712, the embedding vectors in the plurality of global windows are clustered to perform speaker identification, and in step 714, one or more speaker identification sequences are presented based on the speaker identification performed in one or more global windows. This may include, for example, the processor 120 of the electronic device 101 or the server 106 executing the global speaker clustering module 210 to cluster the embedding vectors 306 generated for the segments 304 received within the various global windows 402, 406. This may also include the processor 120 or the server 106 of the electronic device 101 generating and outputting the final separation result 214 based on the global clustering operation. The global window here may include at least one global window that includes (but is larger than) another global window.

[0087] At step 716, speaker identifications determined based on different clustering operations in the clustering operation may be adjusted for consistency (if necessary). This may include, for example, the processor 120 of the electronic device 101 or the server 106 performing the process 500 shown in FIG. 5 or using another technique to ensure that speaker identifications generated in different local or global clustering operations always refer to the same speaker. As a specific example, this may include the processor 120 of the electronic device 101 or the server 106 matching entries in the first speaker identification sequence (result 502) with a portion of entries in the second speaker identification sequence (result 504), which may be based on a timestamp in some cases. This may also include the processor 120 of the electronic device 101 or the server 106 replacing a portion of entries in the second speaker identification sequence with entries in the first speaker identification sequence, and possibly updating the new entries in the second speaker identification sequence to produce a modified speaker identification sequence (result 506).

[0088] Although FIG. 7 shows one example of a method 700 for online speaker separation using local clustering and global clustering, various changes may be made to FIG. 7. For example, although shown as a series of steps, the various steps in FIG. 7 may overlap, occur in parallel, occur in a different order, or occur any number of times. As a specific example, steps 704-706 may occur repeatedly as more and more audio data 202 is received and processed. While these steps 704-706 occur, steps 708-716 may occur so that clustering of embedding vectors 306 and presentation of speaker identifications may be performed as embedding vectors 306 are generated.

[0089] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

[0090] According to an embodiment of the present disclosure, a method for performing online speaker separation includes: obtaining (702) at least a portion of an audio stream containing speech activity, wherein at least the portion of the audio stream includes a plurality of segments (304); generating (706) an embedding vector (306) representing each of the plurality of segments (304); clustering (708) the embedding vector (306) into one or more clusters within each of local windows (308, 316, 318) to perform speaker identification, wherein different clusters correspond to different speakers; and based on the local windows (316, 318), performing speaker identification on the embedded vectors (306); 08, 316, 318) to present (710) at least one first speaker identification sequence; clustering (712) the embedding vectors (306) into one or more clusters within each of the global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318); and presenting (714) at least one second speaker identification sequence based on the speaker identification performed for each of the global windows (402, 406).

[0091] According to an embodiment of the present disclosure, the method may further include: matching entries in different sequences of a plurality of sequences, wherein the plurality of sequences include at least one first speaker identification sequence and at least one second speaker identification sequence; and replacing at least some entries in one speaker identification sequence of the plurality of sequences with at least some entries in another speaker identification sequence of the plurality of sequences, so that clusters generated in the different local windows (308, 316, 318) or the different global windows (402, 406) correspond to common speakers.

[0092] According to an embodiment of the present disclosure, a method of clustering (708) the embedding vectors (306) within each of the local windows (308, 316, 318) may include: in each of the local windows (308, 316, 318), clustering a subsequent subset (312) of the embedding vectors (306) that is different from a previous subset (310) of the embedding vectors (306) such that the subsequent subset (312) includes (i) the embedding vectors (306) of the previous subset (310) and (ii) at least one additional embedding vector (306) that is not included in the previous subset (310).

[0093] According to an embodiment of the present disclosure, each of the partial windows ( 308 , 316 , 318 ) may partially overlap with another partial window ( 308 , 316 , 318 ).

[0094] According to an embodiment of the present disclosure, the global window (402, 406) includes a first global window (402) and a second global window (406); and the second global window (406) is larger than the first global window (402) and includes the first global window (402).

[0095] According to an embodiment of the present invention, the final global window in the global windows (402, 406) may contain the entire audio stream.

[0096] According to an embodiment of the present disclosure, the method may further include: determining similarities between clusters of the embedding vectors (306) in the different local windows (308, 316, 318); and in response to determining that the similarity exceeds a threshold, using a common cluster identifier for the clusters of the embedding vectors (306) in the different local windows (308, 316, 318).

[0097] According to an embodiment of the present disclosure, an apparatus includes: at least one processing device configured to: obtain at least a portion of an audio stream containing speech activity, wherein at least the portion of the audio stream includes a plurality of segments (304); generate an embedding vector (306) representing each of the plurality of segments (304); cluster the embedding vector (306) into one or more clusters within each of local windows (308, 316, 318) to perform speaker identification, wherein different clusters correspond to different speakers; and based on the local windows (308, 316, 318), perform speaker identification on the embedded vectors (306); 16, 318) to initiate presentation of at least one first speaker identification sequence; clustering the embedding vectors (306) into one or more clusters within each of the global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318); and initiating presentation of at least one second speaker identification sequence based on the speaker identification performed for each of the global windows (402, 406).

[0098] According to an embodiment of the present disclosure, at least one processing device may also be configured to: match entries in different sequences of a plurality of sequences, wherein the plurality of sequences include at least one first speaker identification sequence and at least one second speaker identification sequence; and replace at least some entries in one speaker identification sequence of the plurality of sequences with at least some entries in another speaker identification sequence of the plurality of sequences, so that clusters generated in the different local windows (308, 316, 318) or the different global windows (402, 406) correspond to common speakers.

[0099] According to an embodiment of the present disclosure, at least one processing device may further be configured to: cluster a subsequent subset (312) of embedding vectors (306) that is different from a previous subset (310) of embedding vectors (306) in each of the local windows (308, 316, 318) such that the subsequent subset (312) includes (i) the embedding vectors (306) of the previous subset (310) and (ii) at least one additional embedding vector (306) not included in the previous subset (310).

[0100] According to an embodiment of the present disclosure, at least one processing device may also be configured to: determine similarities between clusters of the embedding vectors (306) in the different local windows (308, 316, 318); and in response to determining that the similarity exceeds a threshold, use a common cluster identifier for the clusters of the embedding vectors (306) in the different local windows (308, 316, 318).

[0101] According to an embodiment of the present disclosure, a computer-readable medium containing instructions, when executed by at least one processor: obtain at least a portion of an audio stream containing voice activity, wherein at least the portion of the audio stream includes a plurality of segments (304). The non-transitory computer-readable medium also contains instructions, when executed by at least one processor, for generating an embedding vector (306) representing each of the plurality of segments (304) within each of the local windows (308, 316, 318). The non-transitory computer-readable medium also contains instructions, when executed by at least one processor, for clustering the embedding vector (306) into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers. The non-transitory computer-readable medium further comprises instructions, which when executed by at least one processor, are used to initiate presentation of at least one first speaker identification sequence based on the speaker identification performed for each of the local windows (308, 316, 318); cluster the embedding vectors (306) into one or more clusters within each of the global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318). The non-transitory computer-readable medium further comprises instructions, which when executed by at least one processor, are used to initiate presentation of at least one second speaker identification sequence based on the speaker identification performed for each of the global windows (402, 406).

Claims

1. A method for performing online speaker separation, the method comprising: Obtaining (702) at least a portion of an audio stream containing voice activity, wherein at least the portion of the audio stream comprises a plurality of segments (304); generating (706) an embedding vector (306) representing each segment of the plurality of segments (304); Within each of the local windows (308, 316, 318), clustering (708) the embedding vectors (306) into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers; presenting (710) at least one first speaker identification sequence based on the speaker identification performed for each of the local windows (308, 316, 318); clustering (712) the embedding vectors (306) into one or more clusters within each of global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318); and At least one second speaker identification sequence is presented (714) based on the speaker identification performed for each of the global windows (402, 406).

2. The method according to claim 1, further comprising: matching entries in different sequences of a plurality of sequences, wherein the plurality of sequences includes at least one first speaker identification sequence and at least one second speaker identification sequence; and At least some entries in one speaker identification sequence of the plurality of sequences are replaced with at least some entries in another speaker identification sequence of the plurality of sequences so that clusters generated in the different local windows (308, 316, 318) or the different global windows (402, 406) correspond to common speakers.

3. The method according to claim 1 or 2, wherein: Clustering (708) the embedding vectors (306) within each of the local windows (308, 316, 318) includes: In each of the local windows (308, 316, 318), a subsequent subset (312) of the embedding vectors (306) that is different from a previous subset (310) of the embedding vectors (306) is clustered such that the subsequent subset (312) includes (i) the embedding vectors (306) of the previous subset (310) and (ii) at least one additional embedding vector (306) not included in the previous subset (310).

4. The method according to any one of claims 1 to 3, wherein: Each of the local windows (308, 316, 318) partially overlaps another local window (308, 316, 318).

5. The method according to any one of claims 1 to 4, wherein: The global windows (402, 406) include a first global window (402) and a second global window (406); and The second global window (406) is larger than the first global window (402) and includes the first global window (402).

6. The method according to any one of claims 1 to 5, wherein: A final global window in the global windows (402, 406) contains the entire audio stream.

7. The method according to any one of claims 1 to 6, further comprising: determining similarities between clusters of said embedding vectors (306) in said different local windows (308, 316, 318); as well as In response to determining that the similarity exceeds a threshold, a common cluster identifier is used for the clustering of the embedding vectors (306) in the different local windows (308, 316, 318).

8. An apparatus comprising: At least one processing device configured to: obtaining at least a portion of an audio stream containing voice activity, wherein at least the portion of the audio stream comprises a plurality of segments (304); generating an embedding vector (306) representing each segment of the plurality of segments (304); Within each of the local windows (308, 316, 318), clustering the embedding vectors (306) into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers; initiating presentation of at least one first speaker identification sequence based on said speaker identification performed for each of said local windows (308, 316, 318); clustering the embedding vectors (306) into one or more clusters within each of global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318); and Presentation of at least one second speaker identification sequence is initiated based on the speaker identification performed for each of the global windows (402, 406).

9. The device according to claim 8, wherein: The at least one processing device is further configured to: matching entries in different sequences of a plurality of sequences, wherein the plurality of sequences includes at least one first speaker identification sequence and at least one second speaker identification sequence; and At least some entries in one speaker identification sequence of the plurality of sequences are replaced with at least some entries in another speaker identification sequence of the plurality of sequences so that clusters generated in the different local windows (308, 316, 318) or the different global windows (402, 406) correspond to common speakers.

10. The device according to claim 8 or 9, wherein: The at least one processing device is configured to: A subsequent subset (312) of embedding vectors (306) that is different from a previous subset (310) of embedding vectors (306) is clustered in each of the local windows (308, 316, 318) such that the subsequent subset (312) includes (i) the embedding vectors (306) of the previous subset (310) and (ii) at least one additional embedding vector (306) not included in the previous subset (310).

11. The device according to any one of claims 8 to 10, wherein: Each of the local windows (308, 316, 318) partially overlaps another local window (308, 316, 318).

12. The device according to any one of claims 8 to 11, wherein: The global windows (402, 406) include a first global window (402) and a second global window (406); and The second global window (406) is larger than the first global window (402) and includes the first global window (402).

13. The device according to any one of claims 8 to 12, wherein: A final global window in the global windows (402, 406) contains the entire audio stream.

14. The device according to any one of claims 8 to 13, wherein: The at least one processing device is further configured to: determining similarities between clusters of said embedding vectors (306) in said different local windows (308, 316, 318); as well as In response to determining that the similarity exceeds a threshold, a common cluster identifier is used for the clustering of the embedding vectors (306) in the different local windows (308, 316, 318).

15. A computer readable medium comprising instructions which, when executed, cause at least one processor to: obtaining at least a portion of an audio stream containing voice activity, wherein: At least the portion of the audio stream comprises a plurality of segments (304); generating an embedding vector (306) representing each segment of the plurality of segments (304); Within each of the local windows (308, 316, 318), clustering the embedding vectors (306) into one or more clusters to perform speaker identification, wherein different clusters correspond to different speakers; initiating presentation of at least one first speaker identification sequence based on said speaker identification performed for each of said local windows (308, 316, 318); clustering the embedding vectors (306) into one or more clusters within each of global windows (402, 406) to perform speaker identification, wherein each of the global windows (402, 406) is longer than each of the local windows (308, 316, 318); and Presentation of at least one second speaker identification sequence is initiated based on the speaker identification performed for each of the global windows (402, 406).

Images