A Progressive Contrast Learning Framework for Self-Supervised Speaker Matching

Abstract

The method includes receiving, by a first encoder, an original speech segment; receiving, by a second encoder, an extended speech segment of the original speech segment; generating, by the first encoder, a first speaker expression based on the original speech segment; generating, by the second encoder, a second speaker expression based on the extended speech segment; and generating a contrastive loss based on the first speaker expression and the second speaker expression.

Description

[Technical Field]

[0001] This application claims priority from U.S. Application No. 17 / 734,471, filed with the U.S. Patent and Trademark Office on 2 May 2022, the disclosure thereof is incorporated herein by reference in its entirety.

[0002] This disclosure relates, in general terms, to self-supervised learning systems and methods. [Background technology]

[0003] Speaker verification (SV) aims to determine whether an unknown speech utterance belongs to a specific speaker. Depending on the content constraints, speaker verification can be classified into text-dependent and text-independent speaker verification. Existing methods to enhance speaker representation identification focus on designing effective neural network structures and loss functions. These methods rely on speaker identity labeling for all training utterances. However, annotating large amounts of speech data is costly and expensive. Insufficient labeled training data can limit the performance of SV systems, especially when utterances are recorded on different devices.

[0004] Self-supervised speaker representation learning has gained widespread attention in recent years. Much of the research is based on iterative "clustering-classification" learning frameworks, and their performance is sensitive to a predefined number of clusters. Current self-supervised methods rely on instance identification based on binary classification; that is, each training speech segment is considered an independent class, and training criteria are set to distinguish its own transformed segment from a large number of other speech segments and their extensions. Problems with current clustering-based self-supervised methods include unstable system performance and sensitivity to a predefined number of clusters. Furthermore, estimating the number of clusters is difficult when dealing with large amounts of unlabeled data. Moreover, this conventional cluster update strategy leads to poor generalization performance for unlabeled datasets that have never been seen before. In addition, clustering entire large datasets is time-consuming, and storing their pseudo-labels and cluster centroids requires a large amount of memory. [Overview of the project]

[0005] According to one embodiment of an example, the method may include the steps of: receiving an original audio segment with a first encoder; receiving an extended audio segment of the original audio segment with a second encoder; generating a first speaker representation based on the original audio segment with the first encoder; generating a second speaker representation based on the extended audio segment with the second encoder; and generating a contrast loss based on the first speaker representation and the second speaker representation.

[0006] According to one embodiment of an example, the apparatus may include at least one memory configured to store computer program code, and at least one processor configured to access the computer program code and operate as instructed by the computer program code, wherein the computer program code includes a first receive code configured to cause the at least one processor to receive an original audio segment by a first encoder, a second receive code configured to cause the at least one processor to receive an extended audio segment of the original audio segment by a second encoder, a first generate code configured to cause the at least one processor to generate a first speaker representation based on the original audio segment by the first encoder, a second generate code configured to cause the at least one processor to generate a second speaker representation based on the extended audio segment by the second encoder, and a third generate code configured to cause the at least one processor to generate a contrast loss based on the first and second speaker representations.

[0007] According to one embodiment of an example, a non-temporary computer-readable storage medium can store instructions, and when the instructions are executed by at least one processor of an audio signal reverberation removal processing device, the instructions cause the at least one processor to receive an original audio segment by a first encoder, receive an extended audio segment of the original audio segment by a second encoder, generate a first speaker representation based on the original audio segment by the first encoder, generate a second speaker representation based on the extended audio segment by the second encoder, and generate a contrast loss based on the first speaker representation and the second speaker representation.

[0008] Further embodiments are partially described in the following description, partially revealed therefrom, or can be learned by implementing the present embodiments of this disclosure. [Brief explanation of the drawing]

[0009] The above and other embodiments, features, and aspects of the embodiments of this disclosure will become even clearer from the following description in conjunction with the accompanying drawings. [Figure 1] This is a diagram of a system according to one embodiment. [Figure 2] This is a diagram of the components of the apparatus shown in Figure 1 according to one embodiment. [Figure 3] This is a diagram of a progressive contrastive learning (PCL) framework according to one embodiment. [Figure 4] This is a diagram illustrating the visualization of speaker expression according to one embodiment. [Figure 5] This is a flowchart of a PCL method according to one embodiment. [Modes for carrying out the invention]

[0010] The following detailed description of the embodiments refers to the attached drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0011] Figure 1 is a diagram of a system according to one embodiment. Figure 1 includes a client device 110, a server device 120, and a network 130. The client device 110 and the server device 120 can be interconnected via the network 130, which provides wired connections, wireless connections, or a combination of wired and wireless connections.

[0012] The client device 110 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, server devices, etc.), mobile phones (e.g., smartphones, wireless phones, etc.), camera devices, wearable devices (e.g., a pair of smart glasses or smartwatches), or similar devices.

[0013] The server device 120 may include one or more devices. For example, the server device 120 may be a server device, computing device, or similar device that includes hardware such as a processor and memory, software modules, and combinations thereof, in order to perform the corresponding functions.

[0014] Network 130 may include one or more wired and / or wireless networks. For example, Network 130 may include cellular networks (e.g., fifth-generation (5G) networks, long-term evolution (LTE) networks, third-generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks, ad hoc networks, intranets, the Internet, fiber optic-based networks, or similar, and / or combinations of these or other types of networks.

[0015] The number and configuration of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks with different configurations than those shown in Figure 1. Also, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. In addition, or instead, a set of devices (e.g., one or more devices) may perform one or more functions that are described as being performed by another set of devices.

[0016] Figure 2 shows a diagram of one or more components of the device shown in Figure 1 according to one embodiment. The device 200 shown in Figure 2 may correspond to the client device 110 and / or the server device 120.

[0017] As shown in FIG. 2, the apparatus 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0018] The bus 210 may include components that enable communication between components of the apparatus 200. The processor 220 may be implemented in hardware, software, firmware, or a combination thereof. The processor 220 may be implemented by one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and other types of processing components. The processor 220 may include one or more processors that can be programmed to execute corresponding functions.

[0019] The memory 230 may include a random access memory (RAM), a read only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) for storing information and / or instructions for use by the processor 220.

[0020] The storage component 240 may store information and / or software related to the operation and use of the apparatus 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk (registered trademark), a cartridge, a magnetic tape, and / or other types of non-transitory computer-readable media, along with corresponding drives.

[0021] The input component 250 can include components (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, and / or a microphone) that enable the device 200 to receive information, for example, via user input. The input component 250 may also include sensors (e.g., a Global Positioning System (GPS) component, an accelerometer, a gyroscope, and / or an actuator) that sense information.

[0022] The output component 260 can include components that provide output information from the device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0023] The communication interface 270 can include components such as a transceiver (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 270 can enable the device 200 to receive information from other devices and / or provide information to other devices. For example, the communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a Universal Serial Bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like.

[0024] The device 200 may perform one or more processes described herein. The device 200 may perform operations based on the processor 220 executing software instructions stored in a non-temporary computer-readable medium, such as memory 230 and / or storage component 240. A computer-readable medium is defined herein as a non-temporary memory device. A memory device may include storage space in a single physical storage device or include storage space extended across multiple physical storage devices.

[0025] Software instructions may be read into memory 230 and / or storage component 240 from other computer-readable media or from other devices via the communication interface 270. When the software instructions stored in memory 230 and / or storage component 240 are executed, they may cause the processor 220 to execute one or more processes described herein.

[0026] In addition, or instead, hardwired circuits may be used instead of or in combination with software instructions to perform one or more of the processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuits and software.

[0027] As disclosed herein, a system, method, and apparatus are provided which implement a progressive contrastive learning (PCL) algorithm to dynamically estimate the number of clusters at each step based on the statistical properties of the data itself, rather than a predefined number or generation from the entire dataset, such that the estimated number progressively approaches the ground truth speaker number as the number of steps increases. The system, method, and apparatus may update a data queue with the currently generated augmented samples. For example, a memory queue may be updated with the currently augmented minimum batch data. The system, method, and apparatus may introduce eigendecomposition to estimate the number of speakers in the updated data queue. For example, the affinity matrix of the updated memory queue may be decomposed into eigenvalues ​​to find a latent factor for distinguishing speaker identities. The number of speakers may then be estimated according to the largest eigengic gap at each step. The system, method, and apparatus may assign the queued data to estimated cluster centroids and construct a contrast loss that encourages speaker representations to move closer to their cluster centroids and further away from other cluster centroids. For example, by clustering the queue data using K-means algorithm based on estimated numbers, the diversity of the samples can be compressed into their corresponding factors, while simultaneously forming a control loss that distributes the differences between the samples into other factors. Experimental results on large speaker matching datasets demonstrate the effectiveness of the provided PCL compared to existing self-supervised methods.

[0028] Since negative samples in the queue are collected randomly, the distribution of these samples may not be entirely independent of the query samples, which can lead to many negative pairs that may share similar speaker identity information being forcibly separated. Thus, speaker representations are limited to encoding the unique speaker information of the speech. To mitigate this problem, instead of obtaining prior knowledge of the training data (i.e., an approximate estimate of the total number of speakers), as disclosed herein, the system adds the current expanded minibatch containing speaker vectors extracted from the momentum encoder to a memory queue and removes the oldest minibatch in the queue. The system then processes the updated queue data Q∈R M×D The statistical properties of are its affinity matrix A∈R M×M :A=QQ T The investigation is carried out by calculating the following, where the diagonal elements of A are set to 0. To extract the major latent speaker identity characteristics, the normalized graph Laplacian matrix of affinity matrix A is decomposed into eigenvalues ​​and corresponding eigenvector matrices, and the sorted eigenvalues ​​and eigenvector matrices are, respectively, s1, ..., s M , and S∈R M×M It is expressed as follows. Instead of a predefined number of clusters, the system determines the number of clusters k with the largest eigenvalue gap, as shown in equation (1):

number

[0029] Thus, the number of speakers is dynamically generated from the queued data itself, rather than from appropriately designed hyperparameters based on prior knowledge. The system assigns the queued samples to k clusters using a clustering algorithm such as K-means, and estimates their corresponding centroids. During training, the model can progressively learn more robust speaker representations using increasingly accurate estimations. Therefore, the system can have great generalization ability to handle additional or previously unseen large datasets.

[0030] Rather than implementing the centroid of the clusters generated from the entire dataset, the system implements the centroid at the queue level, which is more portable and effective for investigating speaker characteristics under a self-supervised learning framework. A noise contrastive estimation (NCE) PCL loss function can be formulated as in Equation (2): [Number]

[0031] Here, c i represents the centroid of cluster i to which the extended sample v’ i belongs, is named the positive centroid if so, and the negative centroid otherwise. φ is the dynamically estimated temperature coefficient, which indicates the concentration of speaker representations around the centroid. φ can be set as in Equation (3): [Number]

[0032] Here, Z is the total number of speakers incorporated in this cluster, c is the cluster centroid, α indicates the smoothing constant, which can be set to 10. Further, weights can be used to maintain the characteristics of local smoothness and assist bootstrap clustering as in Equation (4): [Number]

[0033] Figure 3 is a diagram of a PCL framework according to an embodiment. The framework includes an encoder f θ 302, a momentum encoder f θ ' 304, and a speaker estimation block 306. The original audio segment 310 is input to the encoder f θThe extended audio segment of the original audio segment 310 (e.g., extended after the data augmentation process) is supplied to 302 and the momentum encoder f θ '304 is supplied with, respectively, D-dimensional unit norm speaker embedding vectors / representations v∈R 1×D 314 and v'∈R 1×D 316 is generated. The number of speakers in the updated memory queue 318 is estimated using a spectral clustering algorithm. As described above, a control loss 350 is constructed based on the generated pseudolabels in the queue using the affinity matrix 320, eigendecomposition 322, and cluster count 324. Thus, the system encourages query samples to be closer to their corresponding cluster centroids compared to the centroids of other clusters.

[0034] Table 1 shows the results of enhanced channel attention, propagation, and aggregation in the time-delayed neural network (ECAPA-TDNN) with PCL disclosed herein and other self-supervised learning (SSL) SV systems for different types of purposes, including MoCo, MOBY (MoCo+BYOL), and ProtoNCE. Other self-supervised SV systems are listed for comparison. [Table 1]

[0035] Figure 4 shows a visualization of speaker representations according to one embodiment. Figure 4 shows t-distributed stochastic neighbor embedding (T-SNE) visualizations of speaker representations using different SSL frameworks, such as MoCo 402, ProtoNCE 404, and PCL 406 disclosed herein. Different shading indicates different speakers.

[0036] The PCL-optimized SV system disclosed herein learns to create tighter speaker clusters and improve the degree of separation between speakers. This suggests that incremental contrast learning based on data characteristics can generate more distinguishable speaker representations and is robust to channel and noise variations.

[0037] Figure 5 is a flowchart of the PCL method according to one embodiment. In operation 502, the system receives the original speech segment with the first encoder. In operation 504, the system receives the extended speech segment of the original speech segment with the second encoder. In operation 506, the system generates a first speaker representation based on the original speech segment with the first encoder. In operation 508, the system generates a second speaker representation based on the extended speech segment with the second encoder. In operation 510, the system generates a contrast loss based on the first and second speaker representations.

[0038] The foregoing disclosures are illustrative and illustrative, and are not intended to be exhaustive or to limit implementations to the disclosed forms. Modifications and alterations are possible in light of the above disclosures or can be learned from the practice of implementation.

[0039] Some embodiments may relate to systems, methods, and / or computer-readable media in any possible level of technical detail integration. The computer-readable media may include (one or more) computer-readable non-temporary storage media having computer-readable program instructions for causing a processor to perform an operation.

[0040] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by instruction execution devices. Computer-readable storage media can be, for example, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks®, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any suitable combination thereof. When used herein, computer-readable storage media are not to be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0041] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or they can be downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers them for storage on a computer-readable storage medium within each computing / processing device.

[0042] Computer-readable program code / instructions for performing an operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or similar, and procedural programming languages ​​such as the "C" programming language or similar. Computer-readable program instructions may be executed as a whole on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute a computer-readable program instruction by personalizing the electronic circuit using state information of the computer-readable program instruction in order to perform an action or operation.

[0043] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine such that instructions executed by the processor of the computer or other programmable data processing device may create means for performing functions / actions defined in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular way such that the computer-readable storage medium storing the instructions may have a manufactured article containing instructions that perform a manner of function / action defined in one or more blocks of a flowchart and / or block diagram.

[0044] Computer-readable program instructions can also be loaded into a computer, other programmable data processing device, or other device, and a series of operational steps may be executed on the computer, other programmable device, or other device to generate a computer implementation process, so that the instructions executed on the computer, other programmable device, or other device perform a function / action defined in one or more blocks of a flowchart and / or block diagram.

[0045] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or instruction portion having one or more executable instructions for implementing a specific (one or more) logical function. The method, computer system, and computer-readable media may include additional blocks, fewer blocks, different blocks, or blocks configured differently from those shown in the figures. In some alternative implementations, the functions mentioned in the blocks may be performed in a different order than those mentioned in the figures. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs a defined function or action, or a combination of dedicated hardware and computer instructions.

[0046] It is clear that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not an limitation of implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it should be understood that software and hardware can be designed to implement the systems and / or methods described herein.

[0047] Any element, action, or command used herein should not be interpreted as important or essential unless explicitly stated otherwise. Furthermore, when used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, when used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with “one or more.” When only one item is intended, the term “one” or a similar word should be used. Furthermore, when used herein, the terms “have,” “possess,” “have,” etc., are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based” unless explicitly stated otherwise.

[0048] While descriptions of various aspects and embodiments have been presented for illustrative purposes, they are not intended to be exhaustive or to limit the disclosed embodiments. Where certain combinations of multiple mechanisms are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these mechanisms can be combined in ways not specifically described in the claims and / or disclosed in the specification. Each of the dependent claims listed below may depend directly on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with any other claims in the claim set. Numerous changes and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been chosen to best describe the principles, practical applications, or technical improvements to the technologies available on the market, or to enable other those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for speaker identification and matching, which is performed by at least one processor, This is a step to train a dynamic clustering model for speaker identification and matching. The first encoder receives multiple original audio segments, The steps include receiving a plurality of extended audio segments based on the plurality of original audio segments using a second encoder, The first encoder generates a first speaker representation based on the plurality of original speech segments, The steps include generating a second speaker representation based on the plurality of extended speech segments using the second encoder, Prior to evaluating the cluster for the second speaker representation, the steps include inputting the second speaker representation into a memory queue, The steps include dynamically determining the number of clusters available for classifying the plurality of original speech segments based on the statistical characteristics of the second speaker representation in the memory queue, The steps include assigning a cluster to each of the second speaker expressions in the aforementioned second speaker expressions, wherein the cluster is one of the dynamically determined number of clusters, The steps include generating a contrast loss based on the first speaker representation and the assigned second speaker representation, wherein the contrast loss is based on the queue-level centroid associated with the dynamically determined number of clusters, rather than the dataset-level centroid; The training steps include, The steps include applying the first speech as input to the trained dynamic clustering model for speaker identification and matching to match the identity of the speaker of the first speech, A method of having.

2. The method according to claim 1, wherein the second encoder is a momentum encoder.

3. The method according to claim 1, wherein the step of dynamically determining the number of clusters available for classification comprises updating the memory queue based on the second speaker representation.

4. The method according to claim 3, wherein the step of dynamically determining the number of clusters available for classification further comprises generating an affinity matrix of the updated memory queue.

5. The method according to claim 4, wherein the step of dynamically determining the number of clusters available for classification further comprises decomposing the affinity matrix into eigenvalues.

6. The method according to claim 5, wherein the step of dynamically determining the number of clusters available for classification further comprises estimating the number of clusters based on the largest eigengap of the eigenvalues.

7. The method according to claim 3, wherein updating the memory queue comprises removing the oldest minibatch from the memory queue.

8. One or more processors, One or more memory locations containing computer programs, It has, The computer program causes one or more processors to perform the method according to any one of claims 1 to 7. Device.

9. A computer program that causes a computer to perform the method described in any one of claims 1 to 7.

Images