An incremental contrastive learning framework for self-supervised speaker verification

Abstract

The method includes receiving, by a first encoder, an original speech segment; receiving, by a second encoder, an extended speech segment of the original speech segment; generating, by the first encoder, a first speaker expression based on the original speech segment; generating, by the second encoder, a second speaker expression based on the extended speech segment; and generating a contrastive loss based on the first speaker expression and the second speaker expression.

Description

[Technical field]

[0001] This application claims priority from U.S. Application No. 17 / 734,471, filed in the U.S. Patent and Trademark Office on May 2, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to self-supervised learning systems and methods. [Background technology]

[0003] Speaker verification (SV) aims to verify whether an unknown speech utterance belongs to a specific speaker. According to content constraints, speaker verification can be classified into text-dependent and text-independent speaker verification. To enhance the identification of speaker expressions, existing methods focus on designing effective neural network structures and loss functions. These methods rely on speaker identity labeling of all utterances for training. However, annotating large-scale speech data is costly and expensive. Insufficient labeled training data can limit the performance of SV systems, especially when the utterances are recorded with different devices.

[0004] Self-supervised speaker representation learning has attracted widespread attention in recent years. Most of the work is based on an iterative “clustering-classification” learning framework, and the performance is sensitive to the predefined number of clusters. Current self-supervised methods rely on binary classification-based instance identification, i.e., each training speech segment is considered as an independent class, and the training criteria are set to distinguish its own transformed segment from many other speech segments and their extensions. The problem with current clustering-based self-supervised methods is that the system performance is unstable and sensitive to the predefined number of clusters. Also, it is difficult to estimate the number of clusters when dealing with large-scale unlabeled data. Moreover, this stereotypical cluster update strategy leads to poor generalization performance for unseen unlabeled datasets. In addition, clustering the entire large dataset is time-consuming, and storing those pseudo-labels and cluster centroids requires a large amount of memory. Summary of the Invention

[0005] According to one aspect of an example embodiment, the method may include receiving, by a first encoder, an original speech segment; receiving, by a second encoder, an extended speech segment of the original speech segment; generating, by the first encoder, a first speaker expression based on the original speech segment; generating, by the second encoder, a second speaker expression based on the extended speech segment; and generating a contrastive loss based on the first speaker expression and the second speaker expression.

[0006] According to an aspect of an example embodiment, an apparatus may include at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code including a first receiving code configured to cause the at least one processor to receive, via a first encoder, an original speech segment; a second receiving code configured to cause the at least one processor to receive, via a second encoder, an extended speech segment of the original speech segment; a first generating code configured to cause the at least one processor to generate, via the first encoder, a first speaker representation based on the original speech segment; a second generating code configured to cause the at least one processor to generate, via the second encoder, a second speaker representation based on the extended speech segment; and a third generating code configured to cause the at least one processor to generate a contrastive loss based on the first speaker representation and the second speaker representation.

[0007] According to an aspect of an example embodiment, a non-transitory computer-readable storage medium may store instructions that, when executed by at least one processor of an audio signal dereverberation processing device, cause the at least one processor to receive, via a first encoder, an original speech segment, receive, via a second encoder, an extended speech segment of the original speech segment, generate, via the first encoder, a first speaker representation based on the original speech segment, generate, via the second encoder, a second speaker representation based on the extended speech segment, and generate a contrastive loss based on the first speaker representation and the second speaker representation.

[0008] Additional aspects will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of illustrative embodiments of the present disclosure. [Brief description of the drawings]

[0009] The above and other aspects, features, and aspects of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a diagram of a system according to one embodiment. [Diagram 2] FIG. 2 is a diagram of components of the device of FIG. 1 according to one embodiment. [Diagram 3] FIG. 1 is a diagram of a progressive contrastive learning (PCL) framework according to one embodiment. [Figure 4] FIG. 2 is a visualization of speaker expressions according to one embodiment. [Diagram 5] 1 is a flow chart of a PCL method according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following detailed description of example embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0011] Figure 1 is a diagram of a system according to one embodiment, including a client device 110, a server device 120, and a network 130. The client device 110 and the server device 120 may be interconnected via the network 130, which may provide a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0012] The client device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server device, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a camera device, a wearable device (e.g., a pair of smart glasses or a smart watch), or a similar device.

[0013] Server device 120 may include one or more devices. For example, server device 120 may be a server device, a computing device, or the like, including hardware, such as a processor and memory, software modules, and combinations thereof, to perform the corresponding functions.

[0014] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad-hoc network, an intranet, the Internet, an optical fiber-based network, or the like, and / or a combination of these or other types of networks.

[0015] The number and configuration of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different configurations of devices and / or networks than those shown in Figure 1. Also, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally, or instead, a set of devices (e.g., one or more devices) may perform one or more functions that are described as being performed by another set of devices.

[0016]

[0023] Figure 2 is a diagram of components of one or more of the devices of Figure 1 according to one embodiment. The device 200 shown in Figure 2 may correspond to the client device 110 and / or the server device 120.

[0017] As shown in FIG. 2, the device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0018] The bus 210 may include components that enable communication between the components of the device 200. The processor 220 is implemented in hardware, software, firmware, or a combination thereof. The processor 220 may be implemented by one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and other types of processing components. The processor 220 may include one or more processors that can be programmed to perform the corresponding functions.

[0019] Memory 230 may include random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that store information and / or instructions for use by processor 220.

[0020] Storage component 240 may store information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid state disks), compact disks (CDs), digital versatile disks (DVDs), floppy disks, cartridges, magnetic tapes, and / or other types of non-transitory computer-readable media along with corresponding drives.

[0021] Input components 250 may include components (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone) that enable device 200 to receive information, such as via user input. Input components 250 may also include sensors (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator) that sense information.

[0022] Output components 260 may include components that provide output information from device 200 (eg, a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0023] Communications interface 270 may include transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 270 may enable device 200 to receive information from and / or provide information to other devices. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

[0024] The device 200 may perform one or more processes described herein. The device 200 may perform operations based on the processor 220 executing software instructions stored in a non-transitory computer-readable medium, such as the memory 230 and / or the storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device may include storage space within a single physical storage device or may include storage space spread across multiple physical storage devices.

[0025] Software instructions may be loaded into memory 230 and / or storage component 240 from other computer readable media or from other devices via communication interface 270. The software instructions stored in memory 230 and / or storage component 240, when executed, may cause processor 220 to perform one or more of the processes described herein.

[0026] Additionally, or instead, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more of the processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0027] As disclosed herein, systems, methods, and devices are provided that are configured to implement a progressive contrastive learning (PCL) algorithm to dynamically estimate the number of clusters at each step based on statistical characteristics of the data itself, instead of generation from a predefined number or the entire dataset, so that the estimated number gradually approaches the ground truth speaker number with increasing steps. The systems, methods, and devices may update a data queue with the current generated augmented samples. For example, a memory queue may be updated with the current augmented minimum batch data. The systems, methods, and devices may introduce an eigendecomposition to estimate the number of speakers in the updated data queue. For example, an affinity matrix of the updated memory queue may be decomposed into eigenvalues ​​to find potential factors for distinguishing speaker identities. Then, the number of speakers may be estimated according to the maximum eigengap at each step. The systems, methods, and devices may assign queued data to estimated cluster centroids and construct a contrastive loss that encourages speaker representations to be closer to the cluster centroid and away from other cluster centroids. For example, based on the estimated number, the cue data can be clustered by K-means to form a contrastive loss that compresses the diversity of samples into their corresponding factors and simultaneously distributes the differences of samples into other factors.Experimental results on a large-scale speaker verification dataset demonstrate the effectiveness of the proposed PCL compared to existing self-supervised methods.

[0028] Since the negative samples in the queue are collected randomly, the distribution of these samples may not be completely independent of the query samples, which may cause many negative pairs that may share similar speaker identity information to be forced apart. Thus, the speaker representation is limited to encoding the inherent speaker information of the speech. To alleviate this problem, instead of having prior knowledge of the training data (i.e., an approximate estimate of the total number of speakers), as disclosed herein, the system adds the current augmented mini-batch containing the speaker vectors extracted from the momentum encoder to a memory queue and removes the oldest mini-batch in the queue. The system then estimates the updated queue data Q∈R M×D The statistical properties of A are expressed by its affinity matrix A∈R. M×M :A=QQ T where the diagonal elements of A are set to 0. To extract the key latent speaker identity features, the normalized graph Laplacian matrix of the affinity matrix A is decomposed into eigenvalue and corresponding eigenvector matrices, and the sorted eigenvalue and eigenvector matrices are denoted as s1,…,s M , and S∈R M×M Instead of a predefined number of clusters, the system determines the number of clusters k that has the largest eigenvalue gap, as in equation (1):

number

[0029] Thus, the number of speakers is dynamically generated from the queued data itself, instead of from well-designed hyperparameters based on prior knowledge. The system assigns the queued samples to k clusters and estimates their corresponding centroids using a clustering algorithm, e.g., K-means. During training, the model can progressively learn more robust speaker representations with increasingly accurate estimates. Thus, the system can have a large generalization ability to additional or unseen large-scale data.

[0030] The system implements cue-level centroids rather than cluster centroids generated from the entire dataset, which is more portable and effective for investigating speaker characteristics under a self-supervised learning framework. The noise contrastive estimation (NCE) PCL loss function can be formulated as in Equation (2):

number

[0031] where c i is the extended sample v' i represents the centroid of the cluster i to which i belongs, named positive centroid, otherwise named negative centroid, and φ is a dynamically estimated temperature coefficient, which suggests the concentration of speaker expressions around the centroid. φ can be set in equation (3) as:

number

[0032] where Z is the total number of speakers included in this cluster, c is the cluster centroid, and α denotes a smoothing constant, which may be set to 10. Furthermore, weights may be used to preserve the property of local smoothness and aid in bootstrap clustering, as in equation (4):

number

[0033] FIG. 3 is a diagram of a PCL framework according to one embodiment. The framework includes an encoder f θ 302, Momentum Encoder f θ ' 304, and speaker estimation block 306. When an original speech segment 310 is input to the encoder f θ302, an extended speech segment of the original speech segment 310 (e.g., extended after a data extension process) is fed to a momentum encoder f θ ' 304 are fed into the D-dimensional unit-norm speaker embedding vector / representation v∈R 1×D 314 and v'∈R 1×D 316 is generated. A spectral clustering algorithm is used to estimate the number of speakers in the updated memory queue 318. As described above, a contrastive loss 350 is constructed based on the generated pseudo-labels in the queue using the affinity matrix 320, eigendecomposition 322, and cluster counts 324. Thus, the system encourages query samples to be closer to their corresponding cluster centroids compared to the centroids of other clusters.

[0034] Table 1 shows the enhanced channel attention, propagation, and aggregation results for the time-delay neural network with PCL (ECAPA-TDNN) disclosed herein and other self-supervised learning (SSL) SV systems with different types of objectives, including MoCo, MOBY (MoCo+BYOL), and ProtoNCE. Other self-supervised SV systems are listed for comparison. [Table 1]

[0035] Figure 4 is a diagram of a visualization of speaker representations according to one embodiment. Figure 4 shows a t-distributed stochastic neighbor embedding (T-SNE) visualization of speaker representations using different SSL frameworks, such as MoCo 402, ProtoNCE 404, and PCL 406 disclosed herein. Different shading indicates different speakers.

[0036] The PCL-optimized SV system disclosed herein learns to create tighter speaker clusters and improves separation between speakers, suggesting that progressive contrastive training based on data characteristics can generate more distinct speaker representations and is robust to channel and noise variations.

[0037] 5 is a flow chart of a method of PCL according to one embodiment. At operation 502, the system receives an original speech segment by a first encoder. At operation 504, the system receives an extended speech segment of the original speech segment by a second encoder. At operation 506, the system generates a first speaker representation based on the original speech segment by the first encoder. At operation 508, the system generates a second speaker representation based on the extended speech segment by the second encoder. At operation 510, the system generates a contrastive loss based on the first speaker representation and the second speaker representation.

[0038] The foregoing disclosure provides illustration and description, and is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0039] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail. The computer-readable media may include computer-readable non-transitory storage medium(s) having computer-readable program instructions for causing a processor to perform operations.

[0040] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridges in a groove structure having instructions recorded thereon, and any suitable combination thereof. Computer-readable storage medium, as used herein, is not to be construed as being a transitory signal per se, such as, for example, electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating in a waveguide or other transmission medium (e.g., light pulses traveling in a fiber optic cable), or electrical signals transmitted in wires.

[0041] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium into each computing / processing device, or can be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computing / processing device for storage.

[0042] The computer readable program code / instructions for carrying out the operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, or the like, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or a connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform an aspect or operation.

[0043] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of instructing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer readable storage medium having instructions stored thereon comprises an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0044] The computer readable program instructions may also be loaded into a computer, other programmable data processing device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to generate a computer-implemented process such that the instructions executing on the computer, other programmable device, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0045] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions having one or more executable instructions for implementing a particular logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently configured blocks than those shown in the figures. In some alternative implementations, the functions noted in the blocks may occur out of order relative to the order noted in the figures. For example, two blocks shown one after the other may actually be executed simultaneously or substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or a combination of dedicated hardware and computer instructions.

[0046] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to be a limitation of the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, with the understanding that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0047] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." When only one item is intended, the term "a" or similar language is used. Also, as used herein, the terms "have," "having," "having," and the like are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on" unless expressly stated otherwise.

[0048] The description of various aspects and embodiments has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Although specific combinations of features are recited in the claims and / or disclosed in the specification, such combinations are not intended to limit the disclosure of possible implementations. In fact, many of the features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements to the art found in the marketplace of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method executed by at least one processor, comprising: receiving, by a first encoder, an original speech segment; receiving, by a second encoder, an extended speech segment of the original speech segment; generating, by the first encoder, a first speaker representation based on the original speech segments; generating, by the second encoder, a second speaker representation based on the extended speech segment; generating a contrastive loss based on the first speaker expression and the second speaker expression; The method according to claim 1,

2. The method of claim 1 , wherein the second encoder comprises a momentum encoder.

3. The method of claim 1 , further comprising updating a memory cue based on the second speaker expression.

4. The method of claim 3 , wherein generating the symmetric loss comprises generating an affinity matrix for the updated memory queue.

5. The method of claim 4 , wherein generating the contrastive loss further comprises decomposing the affinity matrix into eigenvalues.

6. The method of claim 5 , wherein generating the contrastive loss further comprises estimating a number of clusters based on a maximum eigengap of the eigenvalues.

7. The method of claim 3 , wherein updating the memory queue comprises removing the oldest mini-batch from the memory queue.

8. one or more processors; one or more memories storing a computer program; having The computer program causes the one or more processors to carry out a method according to any one of claims 1 to 7. Device.

9. A computer program product causing a computer to carry out the method according to any one of claims 1 to 7.

Images