A robust cross-domain speaker verification method using multi-level domain adapters
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-01-15
- Publication Date
- 2026-05-26
Smart Images
Figure CN117831539B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speaker verification, and more particularly to a robust cross-domain speaker verification method using a multi-level domain adapter. Background Technology
[0002] Speaker verification involves verifying a speaker's identity based on their speech features. Significant progress has been made in this task with the introduction of deep neural network-based speaker embedding techniques. Previous research has primarily focused on improving the architecture for extracting high-quality speaker embeddings and optimizing loss functions to enhance discrimination performance. These studies have achieved remarkable success on speaker verification tasks, consistently surpassing previous results on benchmark datasets such as VoxCeleb.
[0003] Despite significant academic progress in speaker verification on these benchmark datasets, practical industrial applications still face numerous challenges. This is because real-world scenarios often involve more complex environments, such as varying environmental conditions, distances, and recording device configurations for different individuals. This complexity highlights the problems of multi-domain or cross-domain speaker verification, where "domain" encompasses various factors, including language content, channels, and acoustic environment. When speaker models encounter data from different domains during training, registration, or testing, performance often degrades due to domain mismatch.
[0004] To address the domain mismatch problem, numerous studies have developed various adaptation techniques, including discrepancy-based alignment and domain adversarial learning. Discrepancy-based alignment aims to minimize the discrepancies between domains and facilitate the learning of domain-invariant representations. However, it relies on well-defined distance metrics and struggles to handle data from multiple domains. On the other hand, domain adversarial learning implicitly reduces the discrepancies between data from different domains through minimizing a two-player game. However, achieving a balance between these two tasks is a major challenge for this approach, significantly increasing the complexity of the training process.
[0005] Therefore, those skilled in the art are dedicated to developing a robust cross-domain speaker verification method using a multi-level domain adapter. This involves refining domain labels to obtain better domain information and incorporating a multi-level domain adapter into the model to improve cross-domain learning performance. Summary of the Invention
[0006] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is the performance degradation of speaker verification due to domain mismatch.
[0007] To achieve the above objectives, the present invention provides a robust cross-domain speaker verification method using a multi-level domain adapter, comprising the following steps:
[0008] Step 1: Refine the domain tags to obtain domain information;
[0009] Step 2: Add a multi-level domain adapter to the model to improve cross-domain learning performance.
[0010] Furthermore, the domain is segmented based on a combination of device and distance.
[0011] Furthermore, a domain classifier is trained using training data with explicit domain labels.
[0012] Furthermore, the domain classifier is used to predict the labels for generating data in unknown domains.
[0013] Furthermore, through domain adapters, the model can autonomously learn and utilize domain-specific information, dynamically integrating domain information with speaker-related information.
[0014] Furthermore, the domain adapter includes a block-level domain adapter and an embedded-level domain adapter.
[0015] Furthermore, the block-level domain adapter operates between different blocks of the model architecture.
[0016] Furthermore, the embedding-level domain adapter operates in the final stage of the model to adjust the embedding of the model output.
[0017] Furthermore, when encountering data from domains not seen during training, a suitable domain adapter is assigned using a domain classifier for adaptation.
[0018] Furthermore, the model training includes the following steps:
[0019] Step 2.1: Train a domain classifier using training data with explicit domain labels, and generate pseudo-labels for the remaining unknown data in the training and test sets;
[0020] Step 2.2: Enhance the pre-trained speaker model by adding a domain adapter, fine-tune the adapter, use speaker labels for speaker classification loss, and use real domain labels for adapter control;
[0021] Step 2.3: During the testing process, speaker embeddings are extracted from the test data using real or predicted domain labels and scored.
[0022] In a preferred embodiment of the present invention, domain mismatch can lead to a significant degrade in the performance of the speaker verification system. The present invention proposes a novel module called a "domain adapter" to mitigate the domain mismatch problem. During training, it can progressively and automatically learn information from each domain and adapt it to speaker-related information. This adaptive mechanism reduces domain-related variations, thereby improving the overall consistency of speaker information.
[0023] Speaker feature generation is a multi-layered process, and domain differences can accumulate layer by layer, causing performance degradation in the final verification stage. The "domain adapter" proposed in this invention can be deployed at multiple layers of the model. Through multi-layered domain adapters, the system can mitigate domain differences at different levels, thereby improving overall performance.
[0024] When encountering data from domains not seen during training, the performance of speaker verification systems degrades significantly. This invention utilizes a domain classifier to achieve effective generalization to unseen domains. The trained domain classifier can generate soft labels for predicting unseen domains, which are then used for model training and inference, thereby improving performance in unseen domains.
[0025] Compared with the prior art, the present invention has the following obvious substantive features and significant advantages:
[0026] 1. The speaker verification system with added domain adapters in this invention improves performance by 10% to 20% over the baseline system in three trials on the 3D-Speaker dataset.
[0027] 2. The system performance of adding multiple levels of domain adapters is better than that of adding only one type of domain adapter, which proves the complementarity between multiple levels of adapters.
[0028] 3. The speaker verification system equipped with a domain adapter of this invention outperforms the baseline system when encountering data from unseen domains.
[0029] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description
[0030] Figure 1 This is a system framework diagram of a preferred embodiment of the present invention. Detailed Implementation
[0031] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0032] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.
[0033] like Figure 1 As shown, to effectively utilize multi-domain data, the method of this invention is divided into two parts. First, the domain labels are refined to obtain better domain information; second, a multi-level domain adapter is added to the model to improve cross-domain learning performance.
[0034] First, domains are segmented based on a combination of two attributes—device and distance—with the aim of ensuring that each domain contains speech with the same acoustic environment, as device and distance jointly shape the characteristics of the acoustic environment. For example, the same device can produce different acoustic environments at different distances, and conversely, different devices can produce different environments even at the same distance. For data where domain labels are unavailable, this invention trains a domain classifier using training data with explicit domain labels. After training, this classifier can be used to predict labels for generating data in unknown domains, which are then combined with the true labels for use in subsequent system training or inference. Thus, when encountering data from domains not seen during training, this invention can use the domain classifier to assign appropriate domain adapters for adaptation, thereby improving system performance in unseen domains.
[0035] Once the domain labels are acquired, the next challenge lies in effectively integrating domain information into the model training process. A common approach is to extract domain-relevant features from the raw input or intermediate output of the training model and then compensate for their impact on the raw input. However, this method faces two key difficulties: accurate domain information extraction and effective domain knowledge integration. Unlike conventional approaches, this method does not attempt to extract domain information from the input. Instead, it enables the model to autonomously learn and utilize domain-specific information, dynamically integrating domain information with speaker-related information to improve performance. To achieve this, this invention proposes a novel module called a domain adapter.
[0036] To facilitate the model's autonomous learning of domain information, this invention assumes that domain information can be represented by discrete codes stored in a domain codebook, denoted as C. A domain label d can be used as an index to select a specific code, denoted as c. d The selection process can be represented as:
[0037]
[0038] Where N represents the codebook size, i represents the codebook index, and d i These are either binary indicators (i.e., one-hot hard tags) or weight values (i.e., soft tags). To achieve multi-level adaptation, the framework of this invention includes two different types of domain adapters: Block Domain Adapter (BDA) and Embedded Domain Adapter.
[0039] A block-level domain adapter operates between different blocks in the model architecture. Let H be the output of block i. i The block-level domain adapter connects the current layer output with the domain discrete code C through the dense layer f. d In combination, this can be specifically expressed as:
[0040]
[0041] Here H' i Represents the block output after passing through the adapter, C d For domain code, and operators There are two interpretations: channel-level addition and frequency-level addition. During calculation, the domain code C... d It will expand according to the selected dimension. These two interpretations correspond to two different block-level domain adapter structures, denoted as BDA-C and BDA-F respectively. The former combines the domain information with the model output according to the channel dimension, while the latter combines it according to the frequency dimension. The embedding-level domain adapter acts in the final stage of the model, making certain adjustments to the embedding of the model output. By using two dense layers f and g, the embedding z of the model output can be combined with the discrete domain code c. d It is transformed into a new embedding z':
[0042] z'=f(x+g(c d (3)
[0043] By combining these multi-level domain adapters, the model can autonomously learn domain-specific information and further fuse and integrate it with speaker information using linear transformations. This fusion can mitigate variations from different domains, thereby facilitating the extraction of domain-invariant embeddings.
[0044] The training process of the model can be summarized as follows: 1) Train a domain classifier using training data with explicit domain labels, and generate pseudo-labels for the remaining unknown data in the training and test sets. 2) Enhance the pre-trained speaker model by adding domain adapters, fine-tune the adapters, use speaker labels for speaker classification loss, and use real domain labels for adapter control. 3) During testing, extract speaker embeddings from the test data using real or predicted domain labels and score them.
[0045]
[0046] This invention validated the proposed method in three experiments with a 3D-Speaker. The results show that the system with added block-level and embedded-level domain adapters significantly improved performance compared to the baseline system in all experiments, demonstrating that the domain adapters improve performance and mitigate domain differences. Furthermore, the system combining multi-level domain adapters (block-level and embedded-level) achieves even better results. Simultaneously, these datasets include data from unseen domains, whose labels are generated using a domain classifier. It can be seen that the speaker verification system equipped with domain adapters outperforms the baseline system when encountering data from unseen domains.
[0047] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A robust cross-domain speaker verification method using a multi-level domain adapter, characterized in that, Includes the following steps: Step 1: Refine the domain tags to obtain domain information; Step 2: Add a multi-level domain adapter to the model to improve cross-domain learning performance; Through domain adapters, models can autonomously learn and utilize domain-specific information, dynamically integrating domain information with speaker-related information. The domain adapters include block-level domain adapters and embedded-level domain adapters; The block-level domain adapter operates between different blocks of the model architecture; The embedding-level domain adapter operates in the final stage of the model, adjusting the embedding of the model output.
2. The robust cross-domain speaker verification method using a multi-level domain adapter as described in claim 1, characterized in that, Domains are segmented based on a combination of device and distance.
3. The robust cross-domain speaker verification method using a multi-level domain adapter as described in claim 1, characterized in that, Train a domain classifier using training data with explicit domain labels.
4. The robust cross-domain speaker verification method using a multi-level domain adapter as described in claim 3, characterized in that, The domain classifier is used to predict the labels for generating data in unknown domains.
5. The robust cross-domain speaker verification method using a multi-level domain adapter as described in claim 1, characterized in that, When encountering data from a domain not seen during training, use a domain classifier to assign an appropriate domain adapter for adaptation.
6. The robust cross-domain speaker verification method using a multi-level domain adapter as described in claim 1, characterized in that, The model training includes the following steps: Step 2.1: Train a domain classifier using training data with explicit domain labels, and generate pseudo-labels for the remaining unknown data in the training and test sets; Step 2.2: Enhance the pre-trained speaker model by adding a domain adapter, fine-tune the adapter, use speaker labels for speaker classification loss, and use real domain labels for adapter control; Step 2.3: During the testing process, speaker embeddings are extracted from the test data using real or predicted domain labels and scored.