Self-supervised adversarial training domain adaptive speaker confirmation method in complex scene

Abstract

The invention belongs to the technical field of voiceprint recognition, and particularly discloses a self-supervised adversarial training domain adaptive speaker confirmation method in a complex scene, and the method comprises the steps: obtaining a source domain data set and a target domain data set, the target domain data set being a no-label data set obtained through adding noise based on the source domain data set; performing preprocessing and feature extraction on the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and the target domain respectively; performing self-supervised adversarial training on the target model based on the Mel spectrum feature sample; the target model comprises a multi-scale expansion convolution encoder, a speaker classifier and a domain discriminator. According to the method, self-supervised adversarial training is carried out on the target model by combining the Mel spectrum feature samples corresponding to the source domain and the target domain, speaker classification loss and domain discrimination loss can be introduced at the same time to reduce the difference between the source domain and the target domain, the model learns domain invariant knowledge, and the robustness of the model is improved. And accurate speaker recognition can be realized for a complex scene.

Description

Technical Field

[0001] The present application belongs to the field of voiceprint recognition technology, and more specifically, to a self-supervised adversarial training domain adaptive speaker verification method in complex scenarios. Background Art

[0002] With the continuous advancement of signal processing and deep learning technology, the research and development of speaker recognition technology has also been driven. The current speaker recognition system has achieved a recognition accuracy rate close to 100% under certain conditions, especially in a completely ideal laboratory environment. Although deep learning-driven models have shown excellent results on many public corpus datasets, they often find it difficult to overcome the challenges brought about by data acquisition limitations and the complexity of the real world in practical applications. In particular, speaker recognition systems often encounter performance bottlenecks caused by the mismatch between the training environment and the actual application scenario, which in turn affects their actual performance. Summary of the invention

[0003] In view of the defects of the prior art, the purpose of this application is to improve the domain adaptation capability of the model for speaker recognition, so as to achieve accurate speaker recognition in complex scenarios.

[0004] To achieve the above objectives, in a first aspect, the present application provides a method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios, the method comprising:

[0005] Obtain a source domain dataset and a target domain dataset. The target domain dataset is an unlabeled dataset obtained by adding noise based on the source domain dataset.

[0006] Preprocess and extract features of the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain;

[0007] Based on the Mel-spectrogram feature samples corresponding to the source domain and the Mel-spectrogram feature samples corresponding to the target domain, the target model is trained through self-supervised adversarial training.

[0008] The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract the speaker voiceprint embedding features based on Mel spectrum feature samples. The speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain. The domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

[0009] In a possible implementation, the self-supervised adversarial training of the target model based on the Mel spectrum feature samples corresponding to the source domain and the Mel spectrum feature samples corresponding to the target domain includes:

[0010] Input the Mel spectrum feature samples corresponding to the source domain and the Mel spectrum feature samples corresponding to the target domain to the target model, and obtain the speaker classification loss corresponding to the speaker classifier and the domain discrimination loss corresponding to the domain discriminator;

[0011] Determine the target loss based on speaker classification loss and domain discrimination loss;

[0012] Based on the target loss, adjust the model parameters (weights, biases, etc.) of the target model.

[0013] In a possible implementation, determining the target loss based on the speaker classification loss and the domain discrimination loss includes determining the target loss by the following formula:

[0014] L=L clc (x s ,y s )+βL dis (x s , x t );

[0015] Among them, L represents the target loss, L clc represents the speaker classification loss, L dis represents the domain discrimination loss, x s Represents the Mel spectrum feature sample corresponding to the source domain, y s represents the label corresponding to the source domain, x t represents the Mel spectrum feature sample corresponding to the target domain, and β represents L dis Importance to the target model.

[0016] In one possible implementation, the speaker classification loss is determined by the following formula:

[0017]

[0018] Among them, G f represents the multi-scale expanded convolutional encoder, Gc represents the speaker classifier, represents the Mel spectrum feature sample corresponding to the source domain, i represents the sample number, express The corresponding label, D s Represents the dataset consisting of Mel spectrum feature samples corresponding to the source domain, n s Indicates D s The number of samples, L aam Denotes the additive angular margin loss AAMLoss.

[0019] In one possible implementation, the domain discrimination loss is determined by the following formula:

[0020]

[0021] Among them, G f represents the multi-scale dilated convolutional encoder, D g represents the domain discriminator, for or Represents the Mel spectrum feature samples corresponding to the source domain, represents the Mel spectrum feature sample corresponding to the target domain, i represents the sample number, express The corresponding domain label, D = D s ∪D t , D s represents the dataset composed of Mel spectrum feature samples corresponding to the source domain, D t Represents the data set consisting of Mel spectrum feature samples corresponding to the target domain, n = n s +n t , n s Indicates D s The number of samples, n t Indicates D t The number of samples, L aam Denotes the additive angular margin loss AAMLoss.

[0022] In one possible implementation, β is updated by the following formula:

[0023] When the speaker classification accuracy of the current training round is less than that of the previous training round, β is updated by the following formula:

[0024] β=max{β-δ,β min};

[0025] When the speaker classification accuracy of the current training round is greater than or equal to the speaker classification accuracy of the previous training round, and the domain discrimination accuracy of the current training round is less than the domain discrimination accuracy of the previous training round, β is updated by the following formula:

[0026] β=min{β+δ,β max};

[0027] Among them, β min Indicates the preset minimum value, β max Indicates the preset maximum value, δ indicates the preset step size, max indicates the maximum value, and min indicates the minimum value.

[0028] In a possible implementation manner, the step of obtaining the target domain dataset includes:

[0029] Based on the preset signal-to-noise ratio range, the environmental background noise is added to the source domain data to obtain the target domain data set.

[0030] In a second aspect, the present application provides a self-supervised adversarial training domain adaptive speaker verification device in complex scenarios, comprising:

[0031] The data set acquisition module is used to acquire the source domain data set and the target domain data set. The target domain data set is an unlabeled data set acquired by adding noise based on the source domain data set.

[0032] A preprocessing and feature extraction module is used to preprocess and extract features of the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain;

[0033] A training module, used for performing self-supervised adversarial training on a target model based on Mel-spectrogram feature samples corresponding to a source domain and Mel-spectrogram feature samples corresponding to a target domain;

[0034] The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract the speaker voiceprint embedding features based on Mel spectrum feature samples. The speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain. The domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

[0035] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0037] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0038] (1) Using the MEConformer speaker encoder as the extractor of the speaker voiceprint embedding can simultaneously capture the local dependency and contextual relevance of the voiceprint information, effectively improving the representation ability of the voiceprint embedding to improve the accuracy of speaker recognition.

[0039] (2) A self-supervised adversarial training domain adaptive speaker recognition network (SADAN) for complex scenarios is proposed. By simultaneously introducing speaker classification loss and domain discrimination loss to reduce the difference between the source domain and the target domain, the model learns domain-invariant knowledge, thereby improving the domain adaptation ability of speaker recognition.

[0040] (3) An adaptive loss weight adjustment mechanism is proposed to balance the weights of speaker classification loss and domain discrimination loss in adversarial training, which can ensure that the model maintains the accuracy of speaker recognition while learning domain-invariant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is one of the flow charts of the self-supervised adversarial training domain adaptive speaker verification method in complex scenarios provided by the embodiment of the present application;

[0042] Figure 2 This is the second flow chart of the self-supervised adversarial training domain adaptive speaker verification method in complex scenarios provided by the embodiment of the present application;

[0043] Figure 3 This is a framework diagram of a domain adaptive speaker recognition network based on self-supervised adversarial training provided in an embodiment of the present application;

[0044] Figure 4 It is a structural diagram of a self-supervised adversarial training domain adaptive speaker verification device in a complex scenario provided by an embodiment of the present application;

[0045] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant background knowledge is first introduced as follows.

[0047] As the main way of communication in daily life, voice conversation carries rich voiceprint information, emotional information and content information. Speaker recognition, also known as voiceprint recognition, is a biometric authentication technology that distinguishes the speaker's identity based on the specific voiceprint features contained in the voice. In real life, different speakers can usually be distinguished by timbre. This is because the organs of different human individuals have innate physiological differences, and the acquired development and learning behavior together create an individual's unique pronunciation method. Therefore, each speaker's voiceprint has unique and difficult to imitate characteristics. According to the research on speech signal processing, the sound has short-term stability, so the identity of the speaker can be identified based on the speech signal. The speaker recognition task is an important topic involving speech signal processing, pattern recognition, machine learning and deep learning. The research on its related technologies has both theoretical and practical application significance. Specifically, speaker recognition technology has natural advantages as a biometric recognition technology, which makes it an efficient and secure means of identity authentication in many fields.

[0048] Speaker recognition mainly includes two tasks: speaker identification and speaker confirmation. The former is to identify which person in the registration set belongs to based on the given voiceprint, which is a one-to-many classification problem. The latter is to determine whether it is a given target speaker based on the given voiceprint, which is a one-to-one authentication problem. Depending on whether it is necessary to refer to the content information carried in the voice, speaker recognition can be divided into text-based speaker recognition and text-independent speaker recognition. In the text-based speaker recognition task, it is necessary to input the voice fragment and the corresponding text content at the same time. During the collection process, the user needs to say specific text content, so the user needs to cooperate. The requirements for collecting data sets are relatively difficult, and are mostly suitable for occasions that require high security performance requirements. Text-independent speaker recognition only requires the speaker's voiceprint information without considering the voice content, so the usage scenarios are also more extensive. This application mainly studies the more general text-independent speaker confirmation task.

[0049] With the continuous progress of signal processing and deep learning technology, the research and development of speaker recognition technology has also been driven. At present, speaker recognition systems have achieved a recognition accuracy rate close to 100% under certain conditions, especially in a completely ideal laboratory environment. However, the actual application scenarios are complex and changeable, and the performance of speaker recognition is extremely susceptible to various factors. For example, with age and changes in body shape, the characteristics of an individual's voiceprint will change. There will also be slight differences in the voiceprint of an individual under different health conditions. In addition, unknown noise and diverse channels will also affect the performance of the speaker recognition system. Among them, the main factor that causes the gap between speaker recognition research and application is the domain mismatch problem between training data and test data. Speaker recognition tasks based on deep learning are usually based on the assumption that training data and test data are independent and identically distributed, which is often difficult to meet in reality. For example, the existing speaker recognition system is trained on a quiet Chinese corpus, but is applied to a Chinese noisy scene; or a system trained on an English noisy corpus is applied to a Chinese environment. When there is a difference between the training scene and the application scene, the speaker recognition system will experience a significant performance degradation. This is because the machine learning system cannot guarantee performance on unknown data distributions. In addition, the training data set and the test data set themselves may also be affected by various noise environments, where each sampled environment can be regarded as a potential subdomain. The mismatch of these unknown subdomains will further increase the performance degradation of the speaker recognition system caused by the domain mismatch problem.

[0050] In view of this, researchers are committed to exploring domain adaptation strategies, aiming to accelerate and optimize the model migration process from the resource-rich source domain to the new target domain, ensuring that the model can adapt quickly and maintain high performance. There are three main solutions to the existing domain adaptation methods: one is the domain adaptation model based on the embedding extractor, the second is the correlation alignment adaptation after the embedding, and the third is the back-end domain adaptation method, which is the PLDA score adaptation at the back end of the model. However, the existing domain adaptation speaker recognition tasks still have the following shortcomings:

[0051] (1) First, most of the existing domain adaptive speaker recognition frameworks use speaker voiceprint embedding extractors built based on CNN convolutional neural networks. The extracted embedding features lack the contextual relevance of the voiceprint, resulting in insufficient voiceprint embedding representation capabilities.

[0052] (2) Secondly, the existing domain adaptive speaker recognition framework is too dependent on the label information of the target domain data and requires additional manpower and material resources to collect and label the target domain data. This is time-consuming and labor-intensive and is not suitable for speaker recognition tasks in large-scale new scenarios.

[0053] (3) Finally, the existing domain adaptive speaker recognition framework based on adversarial training ignores the impact of the domain discrimination task on the speaker classification task during the model training process, which causes the model to overemphasize the discrimination of domain knowledge, thereby destroying the distinguishability of voiceprint embedding and causing a decrease in speaker recognition performance.

[0054] In order to overcome the above-mentioned defects, the present application provides a self-supervised adversarial training domain adaptive speaker verification method in complex scenarios, which can realize accurate speaker recognition for complex scenarios.

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0058] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0059] Figure 1 This is one of the flowcharts of the self-supervised adversarial training domain adaptive speaker verification method in complex scenarios provided by the embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101 to S103.

[0060] Step S101, obtaining a source domain dataset and a target domain dataset, where the target domain dataset is an unlabeled dataset obtained by adding noise based on the source domain dataset;

[0061] Step S102, preprocessing and feature extraction are performed on the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain;

[0062] Step S103, performing self-supervised adversarial training on the target model based on the Mel-spectrogram feature samples corresponding to the source domain and the Mel-spectrogram feature samples corresponding to the target domain;

[0063] The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract the speaker voiceprint embedding features based on Mel spectrum feature samples. The speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain. The domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

[0064] The above complex scenarios are explained here. The scenario of identifying the speaker based on quiet (no noise) voiceprint data is a simple scenario, and the scenario of identifying the speaker based on voiceprint data with one or more types of noise is a complex scenario.

[0065] It can be understood that by configuring a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator in the target model, the multi-scale dilated convolutional encoder can be used to extract the speaker voiceprint embedding features, and then the speaker classifier can be used to identify the speaker corresponding to the speaker voiceprint embedding features, and the domain discriminator can be used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain. Combined with the Mel spectrum feature samples corresponding to the source domain and the Mel spectrum feature samples corresponding to the target domain, the target model can be self-supervised adversarial training, and speaker classification loss and domain discrimination loss can be introduced at the same time to reduce the difference between the source domain and the target domain, so that the model learns domain-invariant knowledge, thereby enhancing the model's discrimination robustness, improving the model's domain adaptation ability for speaker recognition, and being able to achieve accurate speaker recognition for complex scenarios.

[0066] For example, Figure 2 This is the second flow chart of the self-supervised adversarial training domain adaptive speaker verification method in complex scenarios provided by the embodiment of the present application, such as Figure 2As shown, the self-supervised adversarial training domain adaptive speaker recognition method proposed in the embodiment of the present application mainly includes two stages: synthesis of target domain data of public corpus and training of domain adaptive speaker recognition framework. The target domain data synthesis process on the public corpus includes the screening of voiceprint corpus and noise corpus, signal-to-noise ratio setting, synthesis mode setting and target domain data synthesis. This application intends to use the Chinese quiet corpus LibriSpeech as the source domain data, and synthesize the target domain data by adding noise in different scenarios. According to literature research, the signal-to-noise ratio interval is usually set to 5-15db. The larger the signal-to-noise ratio, the higher the audio quality. Conversely, the greater the challenge to the algorithm model. From the perspective of noise interference, the speaker recognition system is mainly affected by two types of noise, namely convolution noise and additive noise. Convolution noise is generated by channel distortion, and different sound collection devices undergo convolution operations with the voice signal during voice collection; additive noise usually refers to environmental background noise, and the current environmental sound is added to the voice signal. Additive noise affects speaker recognition systems more frequently. In real life, such noise scenes are ever-changing and ubiquitous, such as noisy human voices on the street, high-decibel car horns, and so on. Therefore, this study intends to use additive noise to synthesize target domain data based on the signal-to-noise ratio. In addition, the training process of the domain adaptive speaker recognition framework includes five steps: audio preprocessing, feature extraction, adaptive model establishment, similarity judgment, and speaker recognition. The first step is to use the synthesized target domain and source domain voiceprint data as input for preprocessing operations such as framing and windowing. All input audio is cut into three-second segments, and the librosa tool is used to extract 80-dimensional Mel-spectrogram features. The extracted source domain features and target domain features are saved in hdf5 files respectively to facilitate data reading during model training. The source domain features and target domain features are then input into the constructed target model, or called the domain adaptive speaker recognition model (corresponding to Figure 2 The adaptive model in the is used for voiceprint embedding extraction, and the model is iteratively optimized through the feedback of the speaker classification loss function AAM-loss and the domain discrimination loss function of the backend.

[0067] The following is an exemplary explanation of the synthesis of target domain data from a public corpus. In order to train a domain adaptive speaker recognition model and explore the effectiveness of the algorithm, this application uses an existing public voiceprint corpus as a source domain dataset, and screens noise corpora in different scenarios to synthesize a target domain dataset by synthesizing additive noise. Among them, the source domain dataset uses the open source quiet voiceprint corpus LibriSpeech-train_clean_360, and the noise corpus selects the office noise in the dataset RNN-Noise and the crowd noise in the dataset FSD50K. Since the selected RNN-Noise and FSD50K noise datasets contain multiple types of noise, 175 and 205 noise audio clips were obtained respectively after screening the office noise and crowd noise. In order to expand the scale of the noise dataset, the screened noise clips were cropped according to a length of 5 seconds, so that a total of 6111 RNN-Noise-Office noise clips and 382 FSD50K-Crowd noise clips were obtained after segmentation. In this application, the signal-to-noise ratio is set to 5DB and 15DB respectively when synthesizing the target domain dataset, so as to observe the adaptability of the domain adaptive speaker recognition model under low signal-to-noise ratio and high signal-to-noise ratio respectively. In addition, in the process of synthesizing the target domain dataset, the length of the quiet audio signal in the source domain is used as the benchmark for signal superposition, and the part with insufficient noise signal adopts a repeated reading and superposition strategy. Finally, the source domain dataset LibriSpeech-train_clean_360, the screened office noise RNN-Noise-Office, the crowd noise FSD50K-Crowd and the synthesized target domain dataset information used in this application are shown in Table 1, where the synthesized target domain datasets are libri-rnno-5db, libri-rnno-15db, libri-fsd50k-5db and libri-fsd50k-15db respectively.

[0068] Table 1 Source domain data, noise data and target domain data information

[0069]

[0070] The following is an exemplary description of the training of a domain adaptive speaker recognition model. The training of a domain adaptive speaker recognition model involves: (1) preprocessing and feature extraction, (2) a domain adaptive speaker recognition model based on self-supervised adversarial training, and (3) experimental results and analysis.

[0071] (1) Preprocessing and feature extraction;

[0072] In the training process of the domain adaptive speaker recognition model, the source domain data and the target domain data must first be preprocessed by framing, windowing, endpoint detection, etc. In addition, all audio data to be input into the deep network model are randomly cropped and data enhanced according to the length of 3 seconds. Then, the librosa tool is used to extract the 80-dimensional Mel-spectrogram feature from the preprocessed audio clips, where the window length FFT in librosa is set to 400 and the distance hop_length between adjacent windows is set to 160.

[0073] (2) Domain adaptive speaker recognition model based on self-supervised adversarial training;

[0074] Speaker recognition technology has achieved excellent performance on public corpora, but the domain mismatch problem in practical applications will cause the performance of the task to drop sharply. The problem of speaker recognition domain adaptation has attracted widespread attention from more and more researchers. Limited by the diversity of application scenarios, it takes a lot of cost to collect and annotate the target domain data. Therefore, this application is committed to using a self-supervised approach to train the model using only source domain data and labels, and no label data from the target domain is required in the entire process.

[0075] Figure 3 This is a diagram of a domain adaptive speaker recognition network framework based on self-supervised adversarial training provided by an embodiment of the present application, such as Figure 3 As shown, this application proposes a domain adaptive speaker recognition network SADAN (S elf-supervised adversarial domain adaption network) based on self-supervised adversarial training based on a multi-scale expanded convolution encoder (Multi-Scale Expand Convolution Transformer, MEConformer) voiceprint embedding extractor to reduce inter-domain differences and learn domain-invariant speaker embedding. However, the introduction of adversarial thinking may cause the model to learn too much domain-invariant knowledge, resulting in a decrease in the speaker recognition effect within the domain. Therefore, an adaptive loss weight adjustment mechanism is introduced in this application to adaptively adjust the size of the β value through the learning effect of the model, thereby balancing the impact of speaker classification tasks and domain discrimination tasks on the overall performance of the model.

[0076] (2-1) Self-supervised domain adversarial training adaptive framework SADAN;

[0077] In the self-supervised adversarial training domain adaptive speaker recognition task proposed in this application, the sample size is n s The labeled source domain dataset can be expressed as And the number of samples without label information is nt The target domain dataset can be expressed as The goal of self-supervised domain adaptation is to learn features that are discriminative and invariant to domain differences, thereby ensuring accurate prediction of unlabeled target data. In the SADAN network, the feature learner (MEConformer), domain adaptation function (domain discriminator), and speaker classifier learning tasks are jointly trained by optimizing the final objective loss function, which can be defined as:

[0078] L=L clc (x s ,y s )+βL dis (x s , x t );

[0079] Where L clc represents the speaker classification loss, L ais Represents the domain discrimination loss, which is a migration loss with multiple possible implementations. β is used to control L dis Importance to the model, L dis It is most commonly used in adversarial loss to encourage the model to learn a domain-invariant feature space through a domain discriminator.

[0080] In the adversarial domain adaptation framework, MEConformer extracts the speaker voiceprint embedding features for the source domain data (Mel-spectrogram feature samples corresponding to the source domain) and outputs the speaker voiceprint embedding features corresponding to the source domain, or called source domain embedding features (Patch_S Embedding). MEConformer extracts the speaker voiceprint embedding features for the target domain data (Mel-spectrogram feature samples corresponding to the target domain) and outputs the speaker voiceprint embedding features corresponding to the target domain, or called target domain embedding features (Patch_T Embedding). Figure 3 As shown in , the source domain embedding features output by MEConformer contain the category tokens corresponding to the source domain data, and the target domain embedding features output by MEConformer contain the category tokens corresponding to the target domain data. Figure 3 As shown, the category tokens corresponding to the source domain data can be input into the speaker classifier, and the speaker classifier identifies the speaker corresponding to the category tokens. Figure 3 As shown, the category tokens corresponding to the source domain data and the category tokens corresponding to the target domain data can be input into the domain discriminator, and the domain discriminator discriminates whether the category tokens belong to the source domain or the target domain.

[0081] This application uses the speaker encoder MEConformer as G fTo learn the embedded features, use a classifier G c To perform speaker classification tasks, and use the domain discriminator D g To perform global feature alignment. Here D g The output state of the category tokens corresponding to the source domain and target domain data. In order to achieve domain knowledge adaptation, in G f and D g The maximum-minimum strategy was adopted: G f Learning domain-invariant features to deceive D g , and D g To distinguish the source domain features from the target domain features, the specific objective function can be expressed as:

[0082]

[0083]

[0084] Where n = n s +n t , D=D s ∪D t , L aam represents AAMLoss (Additive Angular Margin Loss), * can be either s or t to represent the source domain or the target domain, y d Used to indicate domain labels, when y d is 1 pointing to the source domain, when y d When it is 0, it points to the target domain.

[0085] (2-2) Adaptive loss weight adjustment mechanism;

[0086] The introduction of adversarial training ideas may cause the model to learn too much domain knowledge, thus affecting the distinguishability of speaker voiceprint embedding, thereby reducing the performance of speaker classification tasks. Therefore, this application introduces a self-adaptation loss weight adjustment (SLWA) mechanism to adjust the domain adaptation loss weight according to the performance on the validation set, which can ensure that the model maintains the accuracy of speaker recognition while learning domain-invariant features.

[0087] The overall loss function L is defined in the formula of the target loss function above, where β is a hyperparameter that controls the weight of the domain adaptation loss. First, the value of β must be initialized. This application initializes the β value to 0.1. Usually, you should start with a smaller value. During the training process, evaluate the performance of the model on the validation set after each epoch and record the accuracy of speaker classification ACC spk And the accuracy of domain discrimination ACCdom According to ACC spk and ACC dom The value of β is adjusted dynamically according to the changes in , and the strategies adopted are as follows:

[0088] If ACC spk If the value of β decreases, it indicates that the model may have over-adapted to the domain difference. Reduce the value of β to reduce the influence of domain adaptation.

[0089] β=max{β-δ,β min};

[0090] If ACC spk Remain stable or increase slightly, but ACC dom If the value of β decreases, it indicates that the model does a good job in domain adaptation. In this case, the value of β is appropriately increased to further improve domain adaptability.

[0091] β=min{β+δ,β max};

[0092] In order to avoid drastic fluctuations in the value of β, the step size δ is set to 0.1 and the adjustment range is set to 1. min and β max are the minimum and maximum values ​​of β, respectively, that is, the value of β can be adjusted between 0.1 and 1, with a step size of 0.1 for each adjustment. After updating the value of β by evaluating the performance and adaptive strategy on the validation set, continue to train the model until the next checkpoint is evaluated again, and repeat the above steps until the model training is completed.

[0093] (3) Experimental results and analysis;

[0094] This application selects the training set and test set of LibriSpeech-train_clean_360 as the training set of source domain data and the test set of speaker classification task, respectively, and uses the training set and test set of synthetic corpus as the self-supervised training set of target domain data and the final target domain test set, respectively. During the entire training process, the label information of the target domain data is not required. The most popular speaker recognition frameworks MEConformer and ECAPA-TDNN are used as the baseline models of this application for experimental comparison, and the proposed method is subjected to ablation experiments and parameter sensitivity analysis. The experimental environment of this application is NVIDIA GeForce RTX 4090, the number of training rounds is set to 50, the batch size is 32, the initial learning rate is 0.001 and a round-by-round decay strategy is adopted, and the optimizer uses SGD.

[0095] (3-1) Comparative experiments between SADAN and baseline models

[0096] This application compares the proposed model with the baseline models ECAPA-TDNN and MEConformer on the target domain dataset. The two baseline models are trained on the LibriSpeech-train_clean_360 dataset without adaptive training using the original parameter settings. The number of experimental training rounds is set to 50 epochs. The experimental results are shown in Table 2.

[0097] Table 2 Speaker verification accuracy ACC (%) and identity error rate EER (%) of baseline model and SADAN on target domain dataset

[0098]

[0099] According to the experimental results in Table 2, it can be seen that the existing SOTA models MEConformer and ECAPA-TDNN models have a significant decrease in performance when tested on noisy target domain data after being trained on a clean corpus. The SADAN model proposed in this application shows better accuracy on both target domain data sets than the two baseline models. Among them, when the signal-to-noise ratio is 5db and 15db, the constant error rate is reduced by 1.11% and 1.51% respectively compared with the ECAPA-TDNN model. The experimental results effectively verify the effectiveness of the model proposed in this application.

[0100] (3-2) Ablation experiments of SADAN;

[0101] Secondly, in order to verify the effectiveness of the adaptive loss weight adjustment mechanism proposed in this application, ablation experiments were carried out on four target domain data sets. The specific experimental results are shown in Table 3. It can be seen from the experimental results that the adaptive weight loss adjustment mechanism proposed in this application can achieve good results on the target domain data sets, and it has better performance under low signal-to-noise ratio conditions. Compared with not using SLWA, the SADAN model has a maximum improvement of 1.39%. When the model has neither adversarial training AT nor SLWA strategies, the model effect decreases significantly.

[0102] Table 3 Ablation experiment of SADAN on adaptive loss weight adjustment mechanism SLWA

[0103]

[0104] (3-3) SADAN sensitivity analysis;

[0105] Finally, the sensitivity analysis of the parameters of the proposed adaptive weight loss adjustment mechanism is carried out. In the experiment, the influence of the β parameter on the model is explored by setting the weight update step δ to 0.05 and 0.1 respectively. The specific experimental results are shown in Table 4.

[0106] Table 4 Sensitivity analysis of β parameters in SADAN

[0107]

[0108] It can be seen from Table 4 that when the step size δ is set to 0.1, the model can achieve the best performance. When the signal-to-noise ratio of the target domain data set is 5db, the step size is set to 0.1, which is 0.15% and 1.0% lower than the constant error rate of 0.05 on the two target domains, respectively. When the signal-to-noise ratio is 15db, the constant error rate of SADAN decreases by 0.16% and 0.98%, respectively. Therefore, the final step size of the adaptive loss weight in the SADAN proposed in this application is set to 0.1.

[0109] The following is a description of the self-supervised adversarial training domain adaptive speaker verification device in complex scenarios provided by the present application. The self-supervised adversarial training domain adaptive speaker verification device in complex scenarios described below and the self-supervised adversarial training domain adaptive speaker verification method in complex scenarios described above can be referenced to each other.

[0110] Figure 4 is a schematic diagram of the structure of a self-supervised adversarial training domain adaptive speaker verification device in a complex scenario provided by an embodiment of the present application, such as Figure 4 As shown, the device comprises: a data set acquisition module 10, a preprocessing and feature extraction module 20 and a training module 30. Among them:

[0111] The data set acquisition module 10 is used to acquire a source domain data set and a target domain data set, wherein the target domain data set is an unlabeled data set acquired by adding noise based on the source domain data set;

[0112] The preprocessing and feature extraction module 20 is used to preprocess and extract features from the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain;

[0113] A training module 30, configured to perform self-supervised adversarial training on a target model based on the Mel-spectrogram feature samples corresponding to the source domain and the Mel-spectrogram feature samples corresponding to the target domain;

[0114] The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract the speaker voiceprint embedding features based on Mel spectrum feature samples. The speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain. The domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

[0115] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0116] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0117] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0118] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0119] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0120] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0121] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0122] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0123] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0124] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0125] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A self-supervised adversarial training domain adaptive speaker verification method in complex scenarios, characterized in that: include: Acquire a source domain dataset and a target domain dataset, wherein the target domain dataset is an unlabeled dataset obtained by adding noise based on the source domain dataset; Preprocessing and feature extraction are performed on the source domain data set and the target domain data set to obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain; Based on the Mel-spectrogram feature samples corresponding to the source domain and the Mel-spectrogram feature samples corresponding to the target domain, the target model is trained through self-supervised adversarial training. The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract speaker voiceprint embedding features based on Mel spectrum feature samples, the speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain, and the domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

2. According to claim 1, the method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios is characterized by: The self-supervised adversarial training of the target model based on the Mel spectrum feature samples corresponding to the source domain and the Mel spectrum feature samples corresponding to the target domain includes: Inputting the Mel spectrum feature samples corresponding to the source domain and the Mel spectrum feature samples corresponding to the target domain to the target model, and obtaining the speaker classification loss corresponding to the speaker classifier and the domain discrimination loss corresponding to the domain discriminator; Determining a target loss based on the speaker classification loss and the domain discrimination loss; Based on the target loss, mode parameters of the target model are adjusted.

3. According to claim 2, the method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios is characterized in that: The determining the target loss based on the speaker classification loss and the domain discrimination loss includes determining the target loss by the following formula: L=L clc (x s ,y s )+βL dis (x s ,x t ); Wherein, L represents the target loss, L clc represents the speaker classification loss, L dis represents the domain discrimination loss, x s Represents the Mel spectrum feature sample corresponding to the source domain, y s represents the label corresponding to the source domain, x t represents the Mel spectrum feature sample corresponding to the target domain, and β represents L dis Importance to the target model.

4. According to claim 3, the method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios is characterized in that: The speaker classification loss is determined by the following formula: Among them, G f represents the multi-scale dilated convolutional encoder, G c represents the speaker classifier, represents the Mel spectrum feature sample corresponding to the source domain, i represents the sample number, express The corresponding label, D s Represents the dataset consisting of Mel spectrum feature samples corresponding to the source domain, n s Indicates D s The number of samples, L aam Denotes the additive angular margin loss AAMLoss.

5. According to claim 3, the method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios is characterized in that: The domain discrimination loss is determined by the following formula: Among them, G f Denotes the multi-scale dilated convolutional encoder, D g represents the domain discriminator, for or Represents the Mel spectrum feature samples corresponding to the source domain, represents the Mel spectrum feature sample corresponding to the target domain, i represents the sample number, express The corresponding domain label, D = D s ∪D t , D s represents the dataset consisting of Mel spectrum feature samples corresponding to the source domain, D t Represents the data set consisting of Mel spectrum feature samples corresponding to the target domain, n = n s +n t , n s Indicates D s The number of samples, n t Indicates D t The number of samples, L aam Denotes the additive angular margin loss AAMLoss.

6. According to claim 3, the method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios is characterized in that: Update β by the following formula: When the speaker classification accuracy of the current training round is less than that of the previous training round, β is updated by the following formula: β=max{β-δ,β min }; When the speaker classification accuracy of the current training round is greater than or equal to the speaker classification accuracy of the previous training round, and the domain discrimination accuracy of the current training round is less than the domain discrimination accuracy of the previous training round, β is updated by the following formula: β=min{β+δ,β max }; Among them, β min Indicates the preset minimum value, β max Indicates the preset maximum value, δ indicates the preset step size, max indicates the maximum value, and min indicates the minimum value.

7. The method for self-supervised adversarial training domain adaptive speaker verification in complex scenarios according to any one of claims 1 to 6, characterized in that: Obtain the target domain dataset, including: Based on a preset signal-to-noise ratio interval, the environmental background noise is added to the source domain data to obtain the target domain data set.

8. A self-supervised adversarial training domain adaptive speaker verification device in complex scenarios, characterized in that: include: A data set acquisition module, used to acquire a source domain data set and a target domain data set, wherein the target domain data set is an unlabeled data set acquired by adding noise based on the source domain data set; A preprocessing and feature extraction module, used to preprocess and extract features from the source domain data set and the target domain data set, and obtain Mel spectrum feature samples corresponding to the source domain and Mel spectrum feature samples corresponding to the target domain; A training module, used for performing self-supervised adversarial training on a target model based on Mel-spectrogram feature samples corresponding to a source domain and Mel-spectrogram feature samples corresponding to a target domain; The target model includes a multi-scale dilated convolutional encoder, a speaker classifier and a domain discriminator. The multi-scale dilated convolutional encoder is used to extract speaker voiceprint embedding features based on Mel spectrum feature samples, the speaker classifier is used to identify the speaker based on the speaker voiceprint embedding features corresponding to the source domain, and the domain discriminator is used to discriminate whether the speaker voiceprint embedding features belong to the source domain or the target domain.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images