Universal model for voice separation and target voice extraction

Abstract

The invention discloses a universal model for voice separation and target voice extraction, which relates to the field of voice and comprises a separation network, an EDA (electronic design automation) module and a clue network. The separation network is based on a frequency band division recurrent neural network, namely a BSRNN, and is a frequency domain separation network. The EDA module is introduced into a frequency domain speech separation model BSRNN, so that the BSRNN has the capability of processing mixed speech with unknown and variable speaker numbers; and the RawNet3 is used as the clue network to realize a target voice extraction function. The model can be flexibly switched among different tasks according to different inputs, the'universal 'effect is realized, the flexibility and the universality of the model are greatly improved, the additional overhead caused by using different models for different tasks is avoided, and the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of speech, and in particular to a universal model for speech separation and target speech extraction. Background Art

[0002] Real-world sound environments are complex, and speech signals are often contaminated by reverberation, noise, and the voices of other speakers. While the human auditory system is adept at isolating individual sound sources and extracting the target speech in noisy environments, replicating this capability in machines is challenging. Two research areas focus on solving this problem: speech separation (SS) and target speech extraction (TSE). The goal of speech separation is to isolate the clean speech of all speakers from a mixed signal without any prior knowledge of the speakers; the goal of target speech extraction is to extract the clean speech of a specific speaker from a mixed signal using given clues about that speaker.

[0003] With the rapid development of deep learning, deep neural networks have become the mainstream solution for both tasks. Speech separation and target speech extraction share many similarities, such as extracting clean speech from a mixed signal. This makes models used for speech separation generally portable to target speech extraction tasks, such as the TFGridNet model for speech separation and the X-TFGridNet model for target speech extraction, and the Sepformer model for speech separation and the X-Sepformer model for target speech extraction. However, speech separation and target speech extraction have their own unique characteristics: speech separation systems typically require advance knowledge of the number of speakers in the mixed speech and struggle to handle scenarios where the number of speakers changes dynamically; target speech extraction systems need to model the provided clues, extract speaker features, and input these features into the main network. These differences often necessitate the use of different models for different tasks, even if these models may be based on the same main architecture. Therefore, if the similarities between the two tasks can be leveraged while addressing their differences, allowing a common model to be used for both, the system's flexibility will be greatly enhanced, expanding its application scenarios.

[0004] There are many choices for the model structure that serves as the backbone network for general models, ranging from time-domain models such as Conv-TasNet, DPRNN, and SepFormer, to frequency-domain models such as BSRNN and TFGridNet. Time-domain models use learnable adaptive encoders and decoders to directly encode, decode, and model the raw audio signal; frequency-domain models, on the other hand, leverage the short-time invariance of speech signals and use the short-time Fourier transform (STFT) to convert the raw audio signal into a spectrum, where modeling and processing are performed on the spectrum. In recent years, both time-domain and frequency-domain models have developed, and existing models have achieved very good performance. Among them, the Band Split Recurrent Neural Network (BSRNN) is a lightweight, high-performance masking-based frequency-domain model that can be used as the backbone network for general models.

[0005] The clue network is used in target speech extraction to extract the characteristics of the target speaker and output an embedding vector that can represent the characteristics of the target speaker. The speaker embedding network takes the registered speech of the target speaker as input and outputs the corresponding embedding vector. For the same speaker, the speaker embedding network should output an embedding vector with high similarity, while for different speakers, the speaker embedding network should output an embedding vector with high discrimination. Therefore, the embedding vector output by the speaker embedding network can model the speaker characteristics, and this type of network is often used as a clue network in the target speech extraction system. RawNet3 is an advanced end-to-end speaker embedding network that models the speaker's characteristics based on the raw waveform. It has strong performance and can be used as a clue network module in a general model.

[0006] In addition, for situations where the number of speakers in speech separation tasks is unpredictable or may change, previous studies have proposed solutions, such as Deep Clustering and DANet, which are cluster-based; RSAN, OR-PIT, and Conditional TasNet, which iteratively output the speech of each speaker; GatedDPRNN and Multi-Decoder DPRNN, which use separate models or modules for each possible number of speakers; and SepEDA, based on the Encoder-Decoder Attractor (EDA) module. These approaches to solving the speaker number problem and their ultimate performance vary, with the EDA module-based approach being the most prominent. This module can simultaneously estimate the number of speakers and model the characteristics of each speaker in the mixed speech, making it a relatively advanced solution. Summary of the Invention

[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to improve the flexibility of the model.

[0008] To achieve the above objectives, the present invention provides a general model for speech separation and target speech extraction, which is characterized by comprising three parts: a separation network, an EDA module and a clue network; the separation network is based on a band-partitioned recursive neural network, i.e., BSRNN, which is a frequency domain separation network; the EDA module is introduced into the frequency domain speech separation model BSRNN, so that it has the ability to process mixed speech with an unknown and variable number of speakers; RawNet3 is used as the clue network to realize the function of target speech extraction.

[0009] Furthermore, the BSRNN includes three submodules: a frequency band division module, a frequency band and sequence modeling module, and an estimation module.

[0010] Furthermore, the frequency band division module processes the speech signal through the short-time Fourier transform to obtain the complex value spectrum X∈C F×T Divided into multiple specially designed, non-overlapping frequency bands Where F and T represent the number of frames of the spectrum along the frequency dimension and time dimension respectively, K represents the number of frequency bands, and i represents the index of the frequency band, which has different bandwidths G. i The real and imaginary parts of the frequency bands are concatenated and passed through the normalization layer and full connection layer corresponding to each frequency band to obtain the real-valued features corresponding to each frequency band. Where N represents the dimension of the feature, these features will be stacked to obtain a single 3D feature Z∈R N×T×K .

[0011] Furthermore, the band and sequence modeling module takes the feature Z as input and performs alternating sequence-level modeling and band-level modeling; in the sequence-level modeling, RNN is used to model the features along the time dimension T, and the band dimension K is regarded as the batch dimension; and in the band-level modeling, RNN is used to model along the band dimension K, and the time dimension T is regarded as the batch dimension.

[0012] Furthermore, in the sequence-level modeling and the band-level modeling processes, the features first pass through a normalization layer, then a bidirectional LSTM layer, and finally a fully connected layer for restoring the dimension. At the same time, there is a residual connection from the input to the output of each modeling process.

[0013] Furthermore, the general model incorporates the attractors from the EDA module into the features within the frequency band and sequence modeling module, so that the features are divided into multiple branches corresponding to multiple speakers.

[0014] Furthermore, the feature Q∈R processed by the frequency band and sequence modeling module J×N×T×K is fed into the estimation module, where J represents the estimated number of speakers, and the estimation module uses a multi-layer perceptron and stacking operation for each frequency band to obtain the mask M∈C J×F×T and residual R∈C J×F×T ; Finally, the spectrum of the separated speech is Where ⊙ represents the Hadamard product, and then through the inverse short-time Fourier transform iSTFT, the separated speech signal can be obtained.

[0015] Furthermore, in the EDA module, the aggregated features are first input into a long short-term memory network LSTM encoder, and the initial state C of the encoder is e,0 It is a 0 vector, and the state C outputted in the last step e,T It will be used as the initial state C of the LSTM decoder d,0 The decoder takes as input a 0 vector and outputs J+1 vectors, where the first J are attractors that will be incorporated into the features, and J represents the number of speakers in the mixed speech inferred by the EDA module.

[0016] The way J is determined is different in the training phase and the inference phase. In the training phase, J is known in advance, and these J+1 vectors will be fed into a linear-sigmoid layer, which will be trained to predict a sequence of J 1s and the last 0. In the inference phase, vectors are generated one by one until the predicted value of the linear-sigmoid layer is less than 0.5, indicating that the vector currently output by the EDA module is more likely not the desired attractor, and the entire generation process stops.

[0017] Furthermore, RawNet3 is used as the clue network to extract an embedding vector from the registered speech, and the embedding vector models the characteristics of the speaker for the target speech extraction task; and because each of the attractors output from the EDA module also contains the characteristics of a speaker in the mixed speech to a certain extent, that is, the attractor and the output of the clue network have a certain similarity, therefore, when the registered speech is provided, the embedding vector output by the clue network can be used to replace the attractor output by the EDA module to suppress the influence of the interfering speaker and extract the speech of the target speaker, thereby realizing the function of target speech extraction.

[0018] Furthermore, the general model adopts a phased training strategy: the first phase is specifically trained for the speech separation task, and the second phase is fine-tuned for the target speech extraction task based on the model trained in the first phase, while maintaining the speech separation performance as much as possible while obtaining the target speech extraction capability;

[0019] In the first stage of training, the permutation invariant training PIT scheme, which is widely used in speech separation model training, is adopted, and the loss function A negative scale-invariant signal-to-noise ratio (SI-SNR) is selected. In addition, because the model needs to be able to predict the number of speakers in the mixed speech, the linear-sigmoid layer in the EDA module needs to be trained to output the probability of whether the attractor exists. The corresponding binary cross entropy is introduced as the loss function for speaker counting. The final loss function is the weighted sum of the above two: Where α is the weight;

[0020] In the second stage of training, the main network is first initialized with the model weights trained in the first stage, and the clue network is initialized with the pre-trained RawNet3; during the training process, the parameters of the clue network part will be frozen. In order to maintain the performance of the separation task, the output of the clue network and the output of the EDA module are randomly used in the subsequent feature integration at a ratio of 4:1; when using the output of the EDA module, PIT is still used; and when using the output of the clue network, since the order of output in the target speech extraction task needs to correspond to the order of the clue network output, a fixed-order loss function calculation scheme is used to ensure that the output of the model corresponds to the target speaker; in addition, because the subsequent process of integrating the embedding vector into the feature is the same for the output of the attractor and clue networks, and the subsequent processing shares the same set of networks, the attractor and clue network outputs representing the same speaker should be as similar as possible, while the attractor and clue network outputs representing different speakers should be as different as possible; for this reason, a new loss function term consisting of two parts is added:

[0021]

[0022] Where β is the weight of the positive part, which makes the attractor and clue network outputs representing the same speaker as similar as possible; γ is the weight of the negative part, which makes the attractor and clue network outputs representing different speakers as different as possible; represents the loss function of the positive part, Represents the loss function of the negative part; a x and c y Represents the xth attractor and yth clue network output respectively; cosine_similarity(x,y) represents the calculation of the cosine similarity between vector x and vector y. Since the order of speakers represented by attractors is uncertain, it is necessary to permutate them in full like PIT. Find the best permutation φ * Minimize the loss function.

[0023] The present invention has the following technical effects:

[0024] 1. The universal model can perform different tasks based on the input. Specifically, it can perform the target speech extraction task when given clues about the target speaker, and the speech separation task when given no prior information about the target speaker. This greatly increases the model's flexibility and broadens its applicability. While previously, different models had to be designed for different tasks, the universal model only requires a single model.

[0025] 2. The general model uses the EDA module to estimate the number of speakers in mixed speech. Therefore, it can be applied in scenarios where the number of speakers is not given in advance or the number of speakers changes dynamically. This enables the model to perform speech separation tasks normally in scenarios that are more consistent with the real world, greatly improving its practicality.

[0026] 3. Although the general model uses BSRNN as the main body of the network and RawNet3 as the clue network, they are both replaceable modules. In actual application, you can completely choose the appropriate module to replace according to the specific requirements of performance and overhead, while ensuring that the model function remains unchanged.

[0027] 4. Traditional training strategies for speech separation or target speech extraction models are difficult to directly apply to the specific model architecture of general-purpose models. The phased training strategy employed in this invention empowers the model to perform multiple tasks while ensuring that its performance on each task is comparable to similar single-task models.

[0028] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a general model structure of a preferred embodiment of the present invention;

[0030] Figure 2 This is a flow chart of EDA module aggregation in a preferred embodiment of the present invention;

[0031] Figure 3 It is an EDA structure diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0033] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0034] The structure of the general model is shown in the attached Figure 1 As shown, it includes three parts: separation network, EDA module and clue network.

[0035] The separation network part of the general model is based on the Band Split Recurrent Neural Network (BSRNN), which is a frequency domain separation network consisting of three submodules: a band partitioning module, a band and sequence modeling module, and an estimation module.

[0036] The frequency band division module processes the speech signal through short-time Fourier transform to obtain the complex spectrum X∈C F×T Divided into multiple specially designed, non-overlapping frequency bands Where F and T represent the number of frames of the spectrum along the frequency dimension and time dimension respectively, K represents the number of frequency bands, and i represents the index of the frequency bands, which have different bandwidths G. i Subsequently, the real and imaginary parts of these frequency bands are concatenated and passed through the normalization layer and full connection layer corresponding to each frequency band to obtain the real-valued features corresponding to each frequency band. Where N represents the dimension of the feature, these features will be stacked to obtain a single 3D feature Z∈R N×T×K .

[0037] The second module, the Band and Sequence Modeling module, takes feature Z as input and performs alternating sequence-level modeling and band-level modeling. In sequence-level modeling, features are modeled along the time dimension T using an RNN, with the band dimension K treated as the batch dimension. In band-level modeling, features are modeled along the band dimension K using an RNN, with the time dimension T treated as the batch dimension. Specifically, in both modeling passes, features first pass through a normalization layer, then a bidirectional LSTM layer, and finally a fully connected layer for dimensionality recovery. Residual connections are used from the input to the output of each modeling pass.

[0038] Unlike the original BSRNN, the general model incorporates attractors from the EDA module into the features within the second module, splitting the features into multiple branches corresponding to multiple speakers. For example, the EDA module takes the output of the third layer of the second module as input, and the resulting attractors are incorporated into the input of the fourth layer.

[0039] Finally, the feature Q∈R processed by the band and sequence modeling modules J×N×T×K It is fed into the third module, the estimation module, where J represents the estimated number of speakers. This module uses a multi-layer perceptron and stacking operation for each frequency band to obtain the mask M∈C J×F×T and residual R∈C J×F×T Finally, the spectrum of the separated speech is Where ⊙ represents the Hadamard product, and the separated speech signal can be obtained by inverse short-time Fourier transform (iSTFT).

[0040] The input of the EDA module is usually 2D, but the features inside the second module of the separation network are 3D. Therefore, the 3D features are first aggregated into 2D features. The specific process is shown in the attached figure. Figure 2 As shown in Figure 2, the aggregated features are a combination of r = 4 different weighted averages of the low-dimensional projections of the input features, and the weights of these weighted averages are also learnable. In the desired case, these low-dimensional subspaces can represent different aspects of information, such as information about different speakers.

[0041] As attached Figure 3 As shown in the EDA module, the aggregated features are first input into a long short-term memory network (LSTM) encoder, the initial state of which is C e,0 It is a 0 vector, and the state C outputted in the last step e,T It will be used as the initial state C of the LSTM decoder d,0 The decoder takes a zero vector as input and outputs J+1 vectors, where the first J are attractors that will be incorporated into the features (J represents the number of speakers in the mixed speech inferred by the EDA module).

[0042] The method for determining J differs between the training and inference phases. During training, J is known in advance, and these J+1 vectors are fed into a linear-sigmoid layer, which is trained to predict a sequence of J 1s followed by a final 0. During inference, vectors are generated one by one until the linear-sigmoid layer's prediction falls below 0.5 (indicating that the vector currently output by the EDA module is more likely than not the desired attractor, and the generation process ceases).

[0043] Using RawNet3 as the cue network, embedding vectors are extracted from the enrollment speech. These embedding vectors model the characteristics of the speaker and can be used for target speech extraction. Because each attractor output from the EDA module also contains the characteristics of a speaker in the mixed speech to a certain extent, that is, the attractor and the output of the cue network have a certain similarity. Therefore, when provided with enrollment speech, the embedding vector output by the cue network can be used to replace the attractor output by the EDA module to suppress the influence of interfering speakers and extract the speech of the target speaker, achieving the target speech extraction function.

[0044] In addition to the model structure, the training strategy for the general model also differs from that of general speech separation models or target speech extraction models. Because training for both tasks simultaneously is difficult, a phased training strategy is adopted: the first phase is dedicated to training for speech separation, while the second phase fine-tunes the trained model for the target speech extraction task, aiming to achieve the target speech extraction capability while maintaining speech separation performance as much as possible.

[0045] In the first stage of training, the permutation invariant training (PIT) scheme, which is widely used in speech separation model training, is adopted, and the loss function A negative scale-invariant signal-to-noise ratio (SI-SNR) is selected. In addition, because the model needs to be able to predict the number of speakers in the mixed speech, the linear-sigmoid layer in the EDA module needs to be trained to output the probability of whether the current attractor exists. The corresponding binary cross entropy is also introduced here as the loss function for speaker counting. The final loss function is the weighted sum of the above two: Where α is the weight.

[0046] In the second phase of training, the main network is first initialized using the model weights trained in the first phase, and the cue network is initialized using the pre-trained RawNet3. During training, the parameters of the cue network are frozen. To maintain the performance of the separation task, the output of the cue network and the output of the EDA module are randomly allocated in a 4:1 ratio for subsequent feature integration. When using the output of the EDA module, PIT is still used; however, when using the output of the cue network, since the order of the outputs in the target speech extraction task needs to correspond to the order of the cue network outputs, a fixed-order loss function calculation scheme is used to ensure that the model output corresponds to the target speaker. In addition, because the subsequent process of integrating the embedding vectors into the features is the same for the outputs of the attractor and cue networks, and the subsequent processing shares the same set of networks, the outputs of the attractor and cue networks representing the same speaker should be as similar as possible, while the outputs of the attractor and cue networks representing different speakers should be as different as possible. To this end, a new loss function term with two parts is added:

[0047]

[0048] Where β is the weight of the positive part, which makes the attractor and clue network outputs representing the same speaker as similar as possible; γ is the weight of the negative part, which makes the attractor and clue network outputs representing different speakers as different as possible; represents the loss function of the positive part, Represents the loss function of the negative part; a x and c y Represents the xth attractor and yth clue network output respectively; cosine_similarity(x,y) represents the calculation of the cosine similarity between vector x and vector y. Since the order of speakers represented by attractors is uncertain, it is necessary to permutate them in full like PIT. Find the best permutation φ * Minimize the loss function. The preferred embodiments of the present invention are described above in detail. It should be understood that numerous modifications and variations based on the concepts of the present invention can be made by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art based on the concepts of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technologies should fall within the scope of protection defined by the claims.

Claims

1. A general model for speech separation and target speech extraction, characterized in that The method comprises three parts: a separation network, an EDA module, and a clue network. The separation network is based on a band-splitting recursive neural network (BSRNN), which is a frequency-domain separation network. The EDA module is introduced into the frequency-domain speech separation model BSRNN, enabling it to process mixed speech with an unknown and variable number of speakers. RawNet3 is used as the clue network to implement the target speech extraction function.

2. The general model for speech separation and target speech extraction according to claim 1, wherein The BSRNN consists of three submodules: a frequency band division module, a frequency band and sequence modeling module, and an estimation module.

3. The general model for speech separation and target speech extraction as claimed in claim 2, characterized in that The frequency band division module processes the speech signal through short-time Fourier transform to obtain the complex value spectrum X∈C F×T Divided into multiple specially designed, non-overlapping frequency bands Where F and T represent the number of frames of the spectrum along the frequency dimension and time dimension respectively, K represents the number of frequency bands, and i represents the index of the frequency band, which has different bandwidths G. i The real and imaginary parts of the frequency bands are concatenated and passed through the normalization layer and full connection layer corresponding to each frequency band to obtain the real-valued features corresponding to each frequency band. Where N represents the dimension of the feature, these features will be stacked to obtain a single 3D feature Z∈R N×T×K .

4. The general model for speech separation and target speech extraction as claimed in claim 3, wherein: The band and sequence modeling module takes the feature Z as input and performs alternating sequence-level modeling and band-level modeling; in the sequence-level modeling, RNN is used to model the features along the time dimension T, at which time the band dimension K is regarded as the batch dimension; and in the band-level modeling, RNN is used to model along the band dimension K, at which time the time dimension T is regarded as the batch dimension.

5. The general model for speech separation and target speech extraction as claimed in claim 4, characterized in that In the sequence-level modeling and the band-level modeling processes, the features first pass through a normalization layer, then a bidirectional LSTM layer, and finally a fully connected layer for restoring the dimension. At the same time, there is a residual connection from the input to the output of each modeling process.

6. The general model for speech separation and target speech extraction as claimed in claim 5, characterized in that The general model incorporates the attractors from the EDA module into the features within the frequency band and sequence modeling module, so that the features are divided into multiple branches corresponding to multiple speakers.

7. The general model for speech separation and target speech extraction as claimed in claim 6, characterized in that The features Q∈R processed by the frequency band and sequence modeling module J×N×T×K is fed into the estimation module, where J represents the estimated number of speakers, and the estimation module uses a multi-layer perceptron and stacking operation for each frequency band to obtain the mask M∈C J ×F×T and residual R∈C J×F×T ; final , the spectrum of the separated speech is Where ⊙ represents the Hadamard product, and then through the inverse short-time Fourier transform iSTFT, the separated speech signal can be obtained.

8. The general model for speech separation and target speech extraction as claimed in claim 7, wherein: In the EDA module, the aggregated features are first input into a long short-term memory network LSTM encoder, the initial state of the encoder C e,0 It is a 0 vector, and the state C outputted in the last step e,T It will be used as the initial state C of the LSTM decoder d,0 The decoder takes as input a 0 vector and outputs J+1 vectors, where the first J are attractors that will be incorporated into the features, and J represents the number of speakers in the mixed speech inferred by the EDA module. The way J is determined is different in the training phase and the inference phase. In the training phase, J is known in advance, and these J+1 vectors will be fed into a linear-sigmoid layer, which will be trained to predict a sequence of J 1s and the last 0. In the inference phase, vectors are generated one by one until the predicted value of the linear-sigmoid layer is less than 0.5, indicating that the vector currently output by the EDA module is more likely not the desired attractor, and the entire generation process stops.

9. The general model for speech separation and target speech extraction as claimed in claim 8, wherein RawNet3 is used as the clue network to extract embedding vectors from the registered speech. The embedding vectors model the characteristics of the speaker and are used for the target speech extraction task. Because each of the attractors output from the EDA module also contains the characteristics of a speaker in the mixed speech to a certain extent, that is, the attractor and the output of the clue network have certain similarities, when the registered speech is provided, the embedding vectors output by the clue network can be used to replace the attractor output by the EDA module to suppress the influence of the interfering speaker and extract the speech of the target speaker, thereby realizing the function of target speech extraction.

10. The general model for speech separation and target speech extraction according to claim 9, wherein The general model adopts a phased training strategy: the first phase is specifically trained for the speech separation task, and the second phase is fine-tuned for the target speech extraction task based on the trained model in the first phase, while maintaining the speech separation performance as much as possible while achieving the target speech extraction capability; In the first stage of training, the permutation invariant training PIT scheme, which is widely used in speech separation model training, is adopted, and the loss function A negative scale-invariant signal-to-noise ratio (SI-SNR) is selected. In addition, because the model needs to be able to predict the number of speakers in the mixed speech, the linear-sigmoid layer in the EDA module needs to be trained to output the probability of whether the attractor exists. The corresponding binary cross entropy is introduced as the loss function for speaker counting. The final loss function is the weighted sum of the above two: Where α is the weight; In the second stage of training, the main network is first initialized using the model weights trained in the first stage, and the clue network is initialized using the pre-trained RawNet3; During training, the parameters of the clue network will be frozen. In order to maintain the performance of the separation task, the output of the clue network and the output of the EDA module are randomly used in the subsequent feature integration at a ratio of 4:

1. When using the output of the EDA module, PIT is still used. When using the output of the clue network, since the order of output in the target speech extraction task needs to correspond to the order of the clue network output, a fixed-order loss function calculation scheme is used to ensure that the output of the model corresponds to the target speaker. In addition, because the subsequent process of integrating the embedding vector into the feature is the same for the output of the attractor and clue networks, and the subsequent processing shares the same set of networks, the attractor and clue network outputs representing the same speaker should be as similar as possible, while the attractor and clue network outputs representing different speakers should be as different as possible. To this end, a new loss function term consisting of two parts is added: where β is the weight of the positive part, which makes the attractor and clue network outputs representing the same speaker as similar as possible; γ is the weight of the negative part, which makes the attractor and clue network outputs representing different speakers as different as possible; represents the loss function of the positive part, Represents the loss function of the negative part; a x and c y Represents the xth attractor and yth clue network output respectively; cosine_similarity(x,y) represents the calculation of the cosine similarity between vector x and vector y. Since the order of speakers represented by attractors is uncertain, it is necessary to permutate them in full like PIT. Find the best permutation φ * Minimize the loss function.

Images