Robust intelligent synthesized speech speaker verification model training method and system

Abstract

The application provides a kind of robust intelligent synthetic speech speaker confirmation model training method, system, storage medium and electronic equipment, it is related to speech processing technical field.The robust intelligent synthetic speech speaker confirmation model provided by the present application is used to improve the speaker confirmation accuracy of intelligent synthetic speech under strong background noise condition, comprising speech enhancement network, feature extraction network and feature enhancement module.In the training phase of the model, the training set of noisy intelligent synthetic speech data set is transported into speech enhancement network and feature extraction network after STFT feature preprocessing and Fbank feature preprocessing, and the two networks are connected through feature enhancement module for joint training to obtain speaker embedding features with noise-robustness.In the test phase, the model is tested based on the test set of noisy intelligent synthetic speech data set;The optimal model selection is carried out in the mode of repeating the foregoing training and testing alternately until the training iteration number reaches the set maximum value.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and specifically to a training method, system, storage medium, and electronic device for a robust intelligent synthesized speech speaker verification model. Background Technology

[0002] Speaker verification technology falls under the category of speaker recognition technology, and it achieves one-to-one speaker identification. The general process of a deep learning-based end-to-end speaker verification method is as follows: Figure 1 As shown, the two speech samples to be tested obtain their respective speaker-embedding features through a neural network. The obtained speaker-embedding features are scored and compared with a set threshold to obtain the final judgment result.

[0003] In deep learning, three main types of methods are used for speech enhancement in the time domain or time-frequency domain: masking-based, mapping-based, and GAN-based. In the time domain, the speech waveform is directly processed to achieve speech-to-speech enhancement. In the time-frequency domain, the speech is subjected to Fourier transform to obtain time-frequency information or spectrograms, and then enhanced speech is obtained through inverse Fourier transform.

[0004] Currently, patent CN116092470A discloses a method for speaker verification in intelligent synthesized speech based on a Group-Res2Block network. This method uses Group-Res2Block as the main network structure and integrates multi-scale and attention mechanisms to propose a speaker verification network for intelligent synthesized speech, demonstrating good performance on speaker verification tasks in clean speech. However, the speaker verification performance of this method is significantly weakened under strong background noise conditions, indicating poor robustness. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a training method, system, storage medium, and electronic device for a robust intelligent synthesized speech speaker verification model, solving the technical problem of poor robustness under strong background noise conditions.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A training method for a robust intelligent synthesized speech speaker verification model, wherein the intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module; the training method includes:

[0010] S1. Obtain a clean speech dataset and a noisy speech dataset, construct a noisy intelligent synthesized speech dataset, and divide it into a training set and a test set;

[0011] S2, Training Phase:

[0012] The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features.

[0013] The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features.

[0014] The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature;

[0015] The enhanced features are used as input to a feature extraction network to obtain speaker embedding features;

[0016] S3. Testing phase: Model testing is performed based on the test set;

[0017] S4. Repeatedly perform training and testing alternately to select the optimal model until the number of training iterations reaches the set maximum value.

[0018] Preferably, the speech enhancement network comprises, in sequence, a dense residual convolutional encoder, an adaptive feature fusion transport layer, and a dense residual convolutional decoder;

[0019] The dense residual dilated convolutional encoder consists of e dilated convolutional units, each of which is composed of a two-dimensional convolution Conv2D, a normalized layer BN, and an activation function PReLU in sequence; and the first e-1 dilated convolutional units are designed to have dense residual connections.

[0020] The adaptive feature fusion transport layer includes a CBAM adaptive feature fusion module and a Sigmoid-gated adaptive feature fusion module; wherein, the CBAM adaptive feature fusion module includes a DPRNN module, a Conformer module and a CBAM module; the Sigmoid-gated adaptive feature fusion module is composed of a two-dimensional convolution Conv2D, a normalization layer BN, and an activation function Sigmoid in sequence.

[0021] The dense residual convolutional decoder consists of e dilated deconvolutional units, each of which is composed of a two-dimensional deconvolution Conv2DTranspose layer, a normalization layer BN, and an activation function PReLU in sequence; and the first e-1 dilated deconvolutional units are designed with dense residual connections.

[0022] Preferably, it is characterized in that,

[0023] The DPRNN module consists of inter-block RNN ​​units, a reshape function, and intra-block RNN ​​units. The inter-block RNN ​​units sequentially include a Long Short-Term Memory (LSTM) network layer, a fully connected layer (FC), and a layer normalization layer (iLN), and the input features before entering the LSTM are directly added to the output features of the iLN via a residual connection. The intra-block RNN ​​units sequentially include a Bidirectional Long Short-Term Memory (BiLSTM) network layer, a fully connected layer (FC), and a second layer normalization layer (iLN), and the input features before entering the BiLSTM are directly added to the output features of the iLN via a residual connection.

[0024] The Conformer module consists of an intra-block Conformer unit, a reshape function, and an inter-block Conformer unit. The intra-block Conformer unit and the inter-block Conformer unit have the same structure, each consisting of a FeedForword unit, a Multihead-Attention unit, a ConvMoudle unit, and a FeedForward unit. A residual connection is established between the input and output of each unit, which is directly added together.

[0025] The FeedForward unit sequentially includes a layer normalization layer (LN), a fully connected layer (FC), an activation function (Swish), a dropout layer, another fully connected layer (FC), and another dropout layer; the Multihead-Attention unit sequentially includes a layer normalization layer (LN), a multihead-attention layer (Multihead-Attention), and a dropout layer; the ConvMoudle unit sequentially includes a layer normalization layer (LN), a pointwise Conv1D, an activation function (GLU), a depthwise Conv1D, a batch normalization layer (BN), an activation function (Swish), and a pointwise Conv1D.

[0026] The CBAM module is composed of a channel attention mechanism unit and a spatial attention mechanism unit in sequence. The channel attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a multilayer perceptron and an activation function Sigmoid in sequence. The spatial attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a two-dimensional convolution Conv2D, an activation function ReLU, a two-dimensional convolution Conv2D and an activation function Sigmoid in sequence.

[0027] Preferably, during the training phase:

[0028] Based on the CBAM adaptive feature fusion module, the output feature encode_feature of the dense residual dilated convolutional encoder is passed through the DPRNN module and the Conformer module respectively. The output results of the two modules are concatenated on the channel. The concatenated feature is then passed through the CBAM module to obtain the first adaptive fusion feature fusion_feature1.

[0029] Based on the Sigmoid-gated adaptive feature fusion module, the output feature encode_feature of the dense residual dilated convolutional encoder is passed sequentially through the two-dimensional convolution Conv2D, the normalization layer BN, and the activation function Sigmoid. The output is then multiplied with the feature encode_feature to obtain the second adaptive fusion feature fusion_feature2.

[0030] The fusion_feature1 and fusion_feature2 are concatenated on the channel to obtain the final total adaptive fusion feature end_fusion_feature.

[0031] Preferably, the feature enhancement module includes downsampling and two-dimensional convolution Conv2D in sequence.

[0032] Preferably, the feature extraction network includes a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module.

[0033] Preferably, the speech enhancement network and the feature extraction network are linked and jointly trained through the feature enhancement module to construct the following joint loss function:

[0034] L total_loss =L AM-Softmax +β(L SNR +L MSE )

[0035]

[0036]

[0037]

[0038] Among them, L total_loss Represents the joint loss function;

[0039] L AM-Softmax The AM-Softmax loss function of the feature extraction network is represented by ; U represents the total number of noisy speech samples in the training set, m and s are hyperparameters, and f is the sum of the values ​​of m and s. k y represents the speaker embedding features output by the feature extraction network for the k-th training speech sample. k ={1,2,3,...,h} represents the speaker embedding feature f output by the feature extraction network after the k-th training speech sample in the training set is processed. k The speaker's category, where h represents the total number of categories and j represents the category index. Let the y-th weight of the weight matrix W of the feature extraction network be... k Column parameters, T represents transpose, W j This represents the parameter in the j-th column of the weight matrix W, where j = {1, 2, 3, ..., h};

[0040] L SNR Let f(x) represent the signal-to-noise ratio loss function of the speech enhancement network; y represents the clean, noise-free speech; x represents the input noisy speech; f(x) represents the denoised speech predicted by the speech enhancement network; and f(x)-y represents the predicted noise.

[0041] L MSE This represents the mean squared error loss function of the speech enhancement network;

[0042] β is a hyperparameter representing the weight coefficient of the speech enhancement network's loss function in the joint loss function.

[0043] A training system for a robust intelligent synthesized speech speaker verification model, wherein the intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module; the training system includes:

[0044] The acquisition module is used to acquire clean speech datasets and noisy datasets, construct noisy intelligent synthesized speech datasets, and divide them into training sets and test sets;

[0045] The training module is used during the training phase:

[0046] The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features.

[0047] The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features.

[0048] The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature;

[0049] The enhanced features are used as input to a feature extraction network to obtain speaker embedding features;

[0050] The testing module is used during the testing phase to perform model testing based on the test set.

[0051] The optimization module is used to select the optimal model by repeatedly performing training and testing alternately until the number of training iterations reaches the set maximum value.

[0052] A storage medium storing a computer program for training a robust intelligent synthesized speech speaker verification model, wherein the computer program causes a computer to execute the training method for the robust intelligent synthesized speech speaker verification model as described above.

[0053] An electronic device, comprising:

[0054] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a training method for performing a robust intelligent synthesized speech speaker verification model as described above.

[0055] (III) Beneficial Effects

[0056] This invention provides a training method, system, storage medium, and electronic device for a robust intelligent synthesized speech speaker verification model. Compared with existing technologies, it has the following advantages:

[0057] This invention provides a robust intelligent synthesized speech speaker identification model to improve the accuracy of speaker identification in intelligent synthesized speech under strong background noise conditions. The model includes a speech enhancement network, a feature extraction network, and a feature enhancement module. During the model training phase, the training set of the noisy intelligent synthesized speech dataset, after STFT and Fbank feature preprocessing, is fed into the speech enhancement network and feature extraction network respectively. The feature enhancement module then connects the two networks for joint training to obtain speaker embedding features with noise resistance robustness. During the testing phase, the model is tested using the test set of the noisy intelligent synthesized speech dataset. The optimal model is selected by repeatedly performing the aforementioned training and testing alternately until the number of training iterations reaches a set maximum value. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of a general process of an end-to-end speaker verification method based on deep learning in the prior art;

[0060] Figure 2 This is an overall network structure diagram of a robust intelligent synthesized speech speaker verification model provided in an embodiment of the present invention;

[0061] Figure 3 A flowchart illustrating a training method for a robust intelligent synthesized speech speaker verification model provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the structure of a dense residual dilated convolutional encoder provided in an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of the structure of an adaptive feature fusion transport layer provided in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the structure of a DPRNN module provided in an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of the structure of a Conformer module provided in an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of the structure of a CBAM module provided in an embodiment of the present invention;

[0067] Figure 9 This is a schematic diagram of a dense residual dilated convolutional decoder provided in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] This application provides a training method, system, storage medium, and electronic device for a robust intelligent synthesized speech speaker verification model, which solves the technical problem of poor robustness under strong background noise conditions.

[0070] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0071] The robust intelligent synthesized speech speaker verification model provided in this embodiment of the invention is used to improve the speaker verification accuracy of intelligent synthesized speech under strong background noise conditions, such as... Figure 2 As shown, it specifically includes a speech enhancement network, a feature extraction network, and a feature enhancement module.

[0072] During the model training phase, the training set of the noisy intelligent synthesized speech dataset, after STFT feature preprocessing and Fbank feature preprocessing, is fed into the speech enhancement network and feature extraction network, respectively. The two networks are then jointly trained through a feature enhancement module to obtain speaker embedding features with noise resistance. During the testing phase, the model is tested using the test set of the noisy intelligent synthesized speech dataset.

[0073] Repeat the aforementioned alternating training and testing process to select the optimal model until the number of training iterations reaches the set maximum value.

[0074] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0075] Example:

[0076] like Figure 3 As shown, this embodiment of the invention provides a training method for a robust intelligent synthesized speech speaker verification model. The intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module. The training method includes:

[0077] S1. Obtain a clean speech dataset and a noisy speech dataset, construct a noisy intelligent synthesized speech dataset, and divide it into a training set and a test set;

[0078] S2, Training Phase:

[0079] The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features.

[0080] The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features.

[0081] The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature;

[0082] The enhanced features are used as input to a feature extraction network to obtain speaker embedding features;

[0083] S3. Testing phase: Model testing is performed based on the test set;

[0084] S4. Repeatedly perform training and testing alternately to select the optimal model until the number of training iterations reaches the set maximum value.

[0085] The embodiments of the present invention enhance the confirmation performance of the intelligent synthesized speech speaker system under strong background noise, that is, improve its noise resistance robustness.

[0086] The following section will detail the steps of the aforementioned technical solution, based on the specific architecture of the robust intelligent synthesized speech speaker verification model:

[0087] In step S1, a clean speech dataset and a noisy dataset are obtained, a noisy intelligent synthesized speech dataset is constructed, and it is divided into a training set and a test set.

[0088] The clean speech dataset contains natural speech and intelligently synthesized speech. The natural speech comes from the aishell3 dataset, selecting g speakers, with n natural speech samples from each speaker. The intelligently synthesized speech is based on the selected g speakers, using the FastSpeech2 synthesis system for speech synthesis, with p intelligently synthesized speech samples from each speaker. Furthermore, the clean mixed data (natural speech and intelligently synthesized speech) generated by the g speakers is further subdivided: the clean mixed data from h speakers is used as the clean training dataset, and the clean mixed data from f speakers is used as the clean test dataset. Therefore, the clean training dataset contains a total of U speech samples (U = h × (n + p)).

[0089] In this embodiment of the invention, the number of speakers g is 170, the original number of natural speech segments for each speaker is retained, and the number of intelligently synthesized speech segments p for each speaker is 200. The pure training dataset contains 130 speakers h, and the pure test dataset contains 40 speakers f.

[0090] The noise datasets include Audioset and Freesound. Audioset covers human and animal sounds, musical instrument and music genre sounds, and everyday environmental sounds; Freesound mainly includes man-made noises such as snoring, door opening sounds, dragging sounds, and typing sounds. The two noise datasets are each divided into training set noise and test set noise in an a:b ratio. For example, in this embodiment, the ratio of Audioset to Freesound is selected as 9:1.

[0091] During training, training set noise is randomly added to the clean training dataset to generate a noisy training dataset with a signal-to-noise ratio in the range of -5dB to 5dB. The number of speech data points in the noisy training dataset is consistent with the number of speech data points in the clean training dataset, i.e., U data points.

[0092] For the noisy test dataset, random noise is first selected from each clean test speech in the aishell3 clean test set under different signal-to-noise ratios (SNR), generating noisy test datasets with SNRs of -5dB, 0dB, 5dB, and 10dB, respectively. Noisy test groups are then generated for each noisy dataset. For each level of noisy dataset, v noisy speech samples are randomly selected from each speaker's natural noisy speech sample. For each natural noisy speech sample, w noisy synthesized speech samples from that speaker and w noisy synthesized speech samples from other speakers are randomly selected. This results in z (z = 2 × v × w) test groups for each speaker. If the two speech samples in a test group come from the same speaker, the label is set to 1; otherwise, the label is set to 0. (During testing, the cosine distance is calculated based on the speaker embedding features obtained after the two speech samples in the test group are processed by the network to obtain a score. If the score is greater than a set threshold, it is judged as 1; otherwise, it is judged as 0). Since the aishell3 clean test dataset contains f speakers, a total of Q test group data samples (Q = f × z) are generated for each level of noise, with a total of R (R = 2 × Q).

[0093] In this embodiment of the invention, for each level of noisy dataset, a random number v is selected from each person's natural noisy speech, and for each natural noisy speech, 10 synthesized noisy speech samples from that speaker and 10 random synthesized noisy speech samples from other speakers are randomly selected. Therefore, the number of test groups generated for each speaker is z = 100. Since the aishell3 clean test dataset contains 40 speakers (f), a total of 4000 test data groups are generated for each level of noisiness.

[0094] In step S2, i.e. the training phase:

[0095] S21. Perform feature preprocessing on the noisy speech in the training set to obtain the corresponding STFT features and Fbank features. Specifically:

[0096] For the feature extraction network, Fbank features are preprocessed on the noisy input speech to obtain Fbank features. For the speech enhancement network, Short Time Fourier Transform (STFT) features are preprocessed on the noisy input speech. The Fbank features preserve as many original features as possible; the STFT features, containing real and imaginary parts, are input as two data streams into the speech enhancement network.

[0097] S22. The STFT feature is used as the input of the speech enhancement network. After encoding, the total adaptive fusion feature is obtained based on the intermediate transmission layer of the speech enhancement network. The total adaptive fusion feature is then decoded to obtain the decoded feature.

[0098] like Figure 2 As shown, the speech enhancement network in this embodiment of the invention sequentially includes a dense residual convolutional encoder, an adaptive feature fusion transport layer, and a dense residual convolutional decoder. The advantages of introducing dense residual structures and dilated convolutions into the encoder and decoder respectively include: the dense residual structure can reduce the loss of depth feature information, and the dilated convolution can expand the receptive field, making the extracted feature information richer.

[0099] like Figure 4 As shown, the dense residual dilated convolutional encoder consists of e dilated convolutional units, each of which is composed of a two-dimensional convolution Conv2D, a normalized layer BN, and an activation function PReLU in sequence; and the first e-1 dilated convolutional units are designed as dense residual connections.

[0100] When e=4:

[0101] The specific settings for dilated convolution are as follows: the dilation factors of the five dilated convolutional units (Conv2D) are set to 1, 3, 5, 3, and 1 respectively. Since some feature information is skipped in each dilated convolution, the dilation factors are set in this way to ensure that all feature information is used at least once after all convolution processing.

[0102] The specific settings for dense residuals are as follows: the result of the first dilated convolution unit is incremented with a stride of 1 to establish jump connections to the outputs of the second, third, and fourth dilated convolution units; then the result of the second dilated convolution unit is incremented with a stride of 2 to connect to the convolution result of the fourth dilated convolution unit; and so on, with the basic stride increasing for each dilated convolution unit moved forward, until it is directly connected to the fourth dilated convolution unit with the basic stride.

[0103] The original STFT features are processed by the dense residual convolutional encoder module described above to obtain the encoded features encode_feature.

[0104] like Figure 5 As shown, the adaptive feature fusion transport layer includes two parallel adaptive feature fusion modules: a CBAM adaptive feature fusion module and a Sigmoid-gated adaptive feature fusion module. By connecting the Conformer and DPRNN in parallel, the network's ability to extract local and global feature information can be enhanced; the two adaptive feature fusions optimize the final output features of the output layer. Wherein:

[0105] (1) The CBAM adaptive feature fusion module includes the DPRNN module, the Conformer module and the CBAM module.

[0106] like Figure 6 As shown, the DPRNN module consists of inter-block RNN ​​units, a reshape function, and intra-block RNN ​​units. The inter-block RNN ​​units sequentially include a Long Short-Term Memory (LSTM) network layer, a fully connected layer (FC), and a layer normalization layer (iLN), and the input features before entering the LSTM are directly added to the output features of the iLN via a residual connection. The intra-block RNN ​​units sequentially include a Bidirectional Long Short-Term Memory (BiLSTM) network layer, a fully connected layer (FC), and a second layer normalization layer (iLN), and the input features before entering the BiLSTM are directly added to the output features of the iLN via a residual connection.

[0107] like Figure 7 As shown, the Conformer module consists of an intra-block Conformer unit, a reshape function, and an inter-block Conformer unit in sequence. The intra-block Conformer unit and the inter-block Conformer unit have the same structure, each consisting of a FeedForword unit, a Multihead-Attention unit, a ConvMoudle unit, and a FeedForward unit in sequence. A residual connection is established between the input and output of each unit, which is directly added together.

[0108] The FeedForward unit sequentially includes a layer normalization layer (LN), a fully connected layer (FC), an activation function (Swish), a dropout layer, another fully connected layer (FC), and another dropout layer; the Multihead-Attention unit sequentially includes a layer normalization layer (LN), a multihead-attention layer (Multihead-Attention), and a dropout layer; the ConvMoudle unit sequentially includes a layer normalization layer (LN), a pointwise Conv1D, an activation function (GLU), a depthwise Conv1D, a batch normalization layer (BN), an activation function (Swish), and a pointwise Conv1D.

[0109] like Figure 8 As shown, the CBAM module is composed of a channel attention mechanism unit and a spatial attention mechanism unit in sequence; the channel attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a multilayer perceptron and an activation function Sigmoid in sequence; the spatial attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a two-dimensional convolution Conv2D, an activation function ReLU, a two-dimensional convolution Conv2D and an activation function Sigmoid in sequence.

[0110] The encode_feature is passed through the aforementioned DPRNN module and Conformer module respectively, and the outputs of the two modules are concatenated on the channel. The concatenated features are then passed through the CBAM module to obtain the first adaptive fusion feature fusion_feature1.

[0111] (2) Figure 5 As shown, the Sigmoid-gated adaptive feature fusion module consists of a two-dimensional convolutional layer (Conv2D), a normalization layer (BN), and an activation function (Sigmoid). The Sigmoid function controls the value within the range of 0 to 1, thus controlling the degree to which the encoded features are integrated into the first adaptive fusion feature (fusion_feature1).

[0112] The output feature encode_feature of the dense residual dilated convolutional encoder module is sequentially passed through the aforementioned two-dimensional convolution Conv2D, normalization layer BN, and activation function Sigmoid. Then, the output is multiplied by the feature encode_feature to obtain the second adaptive fusion feature fusion_feature2.

[0113] like Figure 9As shown, the dense residual convolutional decoder consists of e dilated deconvolutional units, each of which is composed of a two-dimensional deconvolution Conv2DTranspose layer, a normalization layer BN, and an activation function PReLU in sequence; and the first e-1 dilated deconvolutional units are designed with dense residual connections.

[0114] When e = 4:

[0115] The specific settings for dilated deconvolution are as follows: the dilation factors of the 2D convolution Conv2D of the 5 dilated convolution units are set to 1, 3, 5, 3, 1 in sequence.

[0116] The specific settings for dense residuals are as follows: the result of the first dilated deconvolution unit is incremented with a stride of 1 to establish jump connections to the outputs of the second, third, and fourth dilated deconvolution units; then the result of the second dilated deconvolution unit is incremented with a stride of 2 to connect to the convolution result of the fourth dilated deconvolution unit; and so on, with the basic stride increasing for each dilated deconvolution unit moved forward, until it stops when it is directly connected to the fourth dilated deconvolution unit with the basic stride.

[0117] The total adaptive fusion feature `end_fusion_feature` is processed by the aforementioned dense residual convolutional decoder module to obtain the decoded feature `decode_feature`. In fact, the features output by the intermediate transport layer are bottleneck features, which possess a certain degree of robustness. Therefore, the resulting total adaptive fusion feature `end_fusion_feature` is robust.

[0118] S23. The total adaptive fusion feature is used as the input of the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature.

[0119] like Figure 2 As shown, the feature enhancement module sequentially includes downsampling and 2D convolution (Conv2D). Downsampling is used to maintain consistency with the Fbank feature shape; 2D convolution (Conv2D) is used to reduce the channel dimension to 2.

[0120] The output of the adaptive feature fusion transport layer module of the speech enhancement network, through which the feature enhancement module obtains the feature result, is concatenated with the Fbank features of the proposed feature extraction network in the channel dimension to obtain the enhanced feature `enhance_feature`. Since the Fbank feature has a channel dimension of 1, and the feature result obtained by the feature enhancement module through which the output of the adaptive feature fusion transport layer module of the speech enhancement network obtains the feature result has a channel dimension of 2, the number of channels after feature concatenation is 3.

[0121] S24. Use the enhanced features as input to the feature extraction network to obtain speaker embedding features.

[0122] like Figure 2 As shown, the feature extraction network includes a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module.

[0123] The enhancement_feature is sequentially passed through the feature fusion module under the multi-scale channel attention mechanism, the Group-Res2Block module, the feature fusion module under the layer attention mechanism, and the output module to obtain the final speaker embedding feature with noise resistance robustness.

[0124] Specifically, in this embodiment of the invention, the speech enhancement network and the feature extraction network are linked and jointly trained through the feature enhancement module to construct the following joint loss function:

[0125] L total_loss =L AM-Softmax +β(L SNR +L MSE )

[0126]

[0127]

[0128]

[0129] Among them, L total_loss Represents the joint loss function;

[0130] L AM-Softmax The AM-Softmax loss function of the feature extraction network is represented by ; U represents the total number of noisy speech samples in the training set, m and s are hyperparameters, and f is the sum of the values ​​of m and s. k y represents the speaker embedding features output by the feature extraction network for the k-th training speech sample. k ={1,2,3,...,h} represents the speaker embedding feature f output by the feature extraction network after the k-th training speech sample in the training set is processed. k The speaker's category, where h represents the total number of categories and j represents the category index. Let the y-th weight of the weight matrix W of the feature extraction network be... k Column parameters, T represents transpose, W j Let represent the parameters in the j-th column of the weight matrix W, where j = {1, 2, 3, ..., h}. It's not difficult to understand that the loss function L... AM-SoftmaxBy optimizing the angle θ to reduce intra-class differences and increase inter-class differences, classification accuracy is improved. In this embodiment of the invention, s is set to 30 and m is set to 0.1.

[0131] L SNR Let f(x) represent the signal-to-noise ratio loss function of the speech enhancement network; y represents the clean, noise-free speech; x represents the input noisy speech; f(x) represents the denoised speech predicted by the speech enhancement network; and f(x)-y represents the predicted noise.

[0132] L MSE Let represent the mean squared error loss function of the speech enhancement network.

[0133] β is a hyperparameter representing the weight coefficient of the loss function of the speech enhancement network in the joint loss function. The reason for introducing this hyperparameter is that in the embodiments of this invention, the speech enhancement network is only used as an auxiliary network, and the main goal is to optimize the feature extraction network. During joint training, a weighting coefficient β is added to the loss function of the speech enhancement network. Experiments have shown that β = 0.5 yields the best results.

[0134] In step S3, i.e. the testing phase: model testing is performed based on the test set.

[0135] In step S4, the optimal model is selected by repeatedly performing training and testing alternately until the number of training iterations reaches the set maximum value.

[0136] When the number of training iterations (epochs) is an integer multiple of the set test interval (gap), the current model and its parameters are saved, and the model is tested on a test set generated from a noisy test dataset. After testing, the training process resumes. When the number of training iterations (epochs) again reaches an integer multiple of the set test interval (gap), the testing phase resumes. This alternating training and testing approach is used to select the optimal model until the number of training iterations reaches the set maximum value. This yields the optimal robust intelligent synthesized speech feature extraction network, which is used to improve the speaker identification accuracy of intelligent synthesized speech under strong background noise conditions.

[0137] In this embodiment of the invention, the maximum number of training iterations (epoch) is set to 200, and the test interval (gap) is set to 10. The training process adopts a preheating training strategy, which helps to alleviate the overfitting phenomenon of the network and improve the accuracy. The specific strategy is as follows: the initial learning rate is set to 0.001, and the learning rate is reduced to 0.95 times the original value after each test. The training ends after 200 iterations.

[0138] To better illustrate the superiority of the robust intelligent synthesized speech speaker verification model provided in the embodiments of the present invention, the following comparative experiments are presented for verification:

[0139] First, it should be noted that EER and minDCF in the comparative experiment are two evaluation metrics, and the lower the value, the better; aishell3 and LibriSpeech are two datasets, where aishell3 is a Chinese dataset and LibriSpeech is an English dataset; -5dB, 0dB, 5dB, and 10dB represent the signal-to-noise ratio, and the lower the signal-to-noise ratio, the greater the noise.

[0140] In addition to the model provided in the embodiments of this invention, two existing speech speaker verification models are introduced: CRN and DPCRN.

[0141] Table 1. Experimental results of noise comparison within the training set area.

[0142]

[0143] Table 2. Experimental Results of Gaussian Noise Comparison

[0144]

[0145] Table 3. Experimental Results of Babble Noise Comparison

[0146]

[0147]

[0148] Conclusion: Referring to Tables 1-3, when adding different types and levels of noise to the aishell3 and LibriSpeech datasets, the EER and minDCF values ​​of different methods decreased as the signal-to-noise ratio increased. However, under different signal-to-noise ratio conditions, the model provided by the embodiments of this invention is the best in both evaluation metrics compared to CRN and DPCRN.

[0149] This invention provides a training system for a robust intelligent synthesized speech speaker verification model. The intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module. The training system includes:

[0150] The acquisition module is used to acquire clean speech datasets and noisy datasets, construct noisy intelligent synthesized speech datasets, and divide them into training sets and test sets;

[0151] The training module is used during the training phase:

[0152] The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features.

[0153] The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features.

[0154] The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature;

[0155] The enhanced features are used as input to a feature extraction network to obtain speaker embedding features;

[0156] The testing module is used during the testing phase to perform model testing based on the test set.

[0157] The optimization module is used to select the optimal model by repeatedly performing training and testing alternately until the number of training iterations reaches the set maximum value.

[0158] This invention provides a storage medium storing a computer program for training a robust intelligent synthesized speech speaker verification model, wherein the computer program causes a computer to execute the training method for the robust intelligent synthesized speech speaker verification model as described above.

[0159] An electronic device, comprising:

[0160] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a training method for performing a robust intelligent synthesized speech speaker verification model as described above.

[0161] It is understood that the training system, storage medium, and electronic device of the robust intelligent synthesized speech speaker verification model provided in the embodiments of the present invention correspond to the training method of the robust intelligent synthesized speech speaker verification model provided in the embodiments of the present invention. The explanation, examples, and beneficial effects of the relevant contents can be referred to the corresponding parts of the training method, and will not be repeated here.

[0162] In summary, compared with existing technologies, it has the following beneficial effects:

[0163] 1. The robust intelligent synthesized speech speaker identification model provided by this invention is used to improve the speaker identification accuracy of intelligent synthesized speech under strong background noise conditions. It includes a speech enhancement network, a feature extraction network, and a feature enhancement module. During the model training phase, the training set of the noisy intelligent synthesized speech dataset, after STFT feature preprocessing and Fbank feature preprocessing, is fed into the speech enhancement network and the feature extraction network, respectively. The feature enhancement module connects the two networks for joint training to obtain speaker embedding features with noise resistance robustness. During the testing phase, the model is tested based on the test set of the noisy intelligent synthesized speech dataset. The optimal model is selected by repeatedly performing the aforementioned training and testing alternately until the number of training iterations reaches a set maximum value.

[0164] 2. The advantages of introducing dense residual structures and dilated convolutions in the encoder and decoder respectively include: dense residual structures can reduce the loss of depth feature information, and dilated convolutions can expand the receptive field, making the extracted feature information richer.

[0165] 3. By connecting Conformer and DPRNN in parallel, the network's ability to extract local and global feature information can be enhanced; the two adaptive feature fusions optimize the final output features of the output layer.

[0166] 4. The feature enhancement module includes downsampling and 2D convolution (Conv2D). Downsampling is used to maintain consistency with the Fbank feature shape; 2D convolution (Conv2D) is used to reduce the channel dimension to 2.

[0167] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a robust intelligent synthesized speech speaker verification model, characterized in that, The intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module; The training method includes: S1. Obtain a clean speech dataset and a noisy speech dataset, construct a noisy intelligent synthesized speech dataset, and divide it into a training set and a test set; S2, Training Phase: The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features. The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features. The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature; The enhanced features are used as input to a feature extraction network to obtain speaker embedding features; S3. Testing phase: Model testing is performed based on the test set; S4. Repeatedly perform training and testing alternately to select the optimal model until the number of training iterations reaches the set maximum value; The speech enhancement network comprises, in sequence, a dense residual convolutional encoder, an adaptive feature fusion transport layer, and a dense residual convolutional decoder. The dense residual convolutional encoder consists of e dilated convolutional units, each of which is composed of a two-dimensional convolutional layer Conv2D, a normalized layer BN, and an activation function PReLU in sequence; and the first e-1 dilated convolutional units are designed as dense residual connections. The adaptive feature fusion transport layer includes a CBAM adaptive feature fusion module and a Sigmoid-gated adaptive feature fusion module; wherein, the CBAM adaptive feature fusion module includes a DPRNN module, a Conformer module and a CBAM module; the Sigmoid-gated adaptive feature fusion module is composed of a two-dimensional convolution Conv2D, a normalization layer BN, and an activation function Sigmoid in sequence. The dense residual convolutional decoder consists of e dilated deconvolutional units, each of which is composed of a two-dimensional deconvolution Conv2DTranspose layer, a normalization layer BN, and an activation function PReLU in sequence; and the first e-1 dilated deconvolutional units are designed with dense residual connections.

2. The training method for the robust intelligent synthesized speech speaker verification model as described in claim 1, characterized in that, The DPRNN module consists of inter-block RNN ​​units, a reshape function, and intra-block RNN ​​units. The inter-block RNN ​​units sequentially include a Long Short-Term Memory (LSTM) network layer, a fully connected layer (FC), and a layer normalization layer (iLN), and the input features before entering the LSTM are directly added to the output features of the iLN via a residual connection. The intra-block RNN ​​units sequentially include a Bidirectional Long Short-Term Memory (BiLSTM) network layer, a fully connected layer (FC), and a second layer normalization layer (iLN), and the input features before entering the BiLSTM are directly added to the output features of the iLN via a residual connection. The Conformer module consists of an intra-block Conformer unit, a reshape function, and an inter-block Conformer unit. The intra-block Conformer unit and the inter-block Conformer unit have the same structure, each consisting of a FeedForword unit, a Multihead-Attention unit, a ConvMoudle unit, and a FeedForward unit. A residual connection is established between the input and output of each unit, which is directly added together. The FeedForward unit sequentially includes a layer normalization layer (LN), a fully connected layer (FC), an activation function (Swish), a dropout layer, another fully connected layer (FC), and another dropout layer; the Multihead-Attention unit sequentially includes a layer normalization layer (LN), a multihead-attention layer (Multihead-Attention), and a dropout layer; the ConvMoudle unit sequentially includes a layer normalization layer (LN), a pointwise Conv1D, an activation function (GLU), a depthwise Conv1D, a batch normalization layer (BN), an activation function (Swish), and a pointwise Conv1D. The CBAM module is composed of a channel attention mechanism unit and a spatial attention mechanism unit in sequence. The channel attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a multilayer perceptron and an activation function Sigmoid in sequence. The spatial attention mechanism unit is composed of parallel max pooling layers and average pooling layers, a two-dimensional convolution Conv2D, an activation function ReLU, a two-dimensional convolution Conv2D and an activation function Sigmoid in sequence.

3. The training method for the robust intelligent synthesized speech speaker verification model as described in claim 2, characterized in that, During the training phase: Based on the CBAM adaptive feature fusion module, the output feature encode_feature of the dense residual convolutional encoder is passed through the DPRNN module and the Conformer module respectively. The output results of the two modules are concatenated on the channel. The concatenated feature is then passed through the CBAM module to obtain the first adaptive fusion feature fusion_feature1. Based on the Sigmoid-gated adaptive feature fusion module, the output feature encode_feature of the dense residual convolutional encoder is sequentially passed through the two-dimensional convolution Conv2D, the normalization layer BN, and the activation function Sigmoid. The output is then multiplied with the feature encode_feature to obtain the second adaptive fusion feature fusion_feature2. The fusion_feature1 and fusion_feature2 are concatenated on the channel to obtain the final total adaptive fusion feature end_fusion_feature.

4. The training method for the robust intelligent synthesized speech speaker verification model as described in claim 1, characterized in that, The feature enhancement module includes downsampling and two-dimensional convolution Conv2D.

5. The training method for the robust intelligent synthesized speech speaker verification model as described in claim 1, characterized in that, The feature extraction network includes a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module.

6. The training method for the robust intelligent synthesized speech speaker verification model as described in claim 1, characterized in that, The speech enhancement network and the feature extraction network are linked through the feature enhancement module and jointly trained to construct the following joint loss function: in, Represents the joint loss function; The AM-Softmax loss function of the feature extraction network is represented; U represents the total number of noisy speech samples in the training set, and m and s are hyperparameters. This represents the speaker embedding features output by the feature extraction network after the k-th training speech sample is processed. This represents the speaker embedding feature output by the feature extraction network after processing the k-th training speech sample in the training set. The speaker's category, where h represents the total number of categories and j represents the category index. The weight matrix w of the feature extraction network represents the first weight of the feature extraction network. Column parameters, T represents transpose, This represents the parameter in the j-th column of the weight matrix w. ; Let represent the signal-to-noise ratio loss function of the speech enhancement network; y represents the clean, noise-free speech, and x represents the input noisy speech. This represents the denoised speech predicted by the speech enhancement network. Indicates the noise in the prediction; This represents the mean squared error loss function of the speech enhancement network; is a hyperparameter representing the weight coefficients of the loss function of the speech enhancement network in the joint loss function.

7. A training system for a robust intelligent synthesized speech speaker verification model, characterized in that, The intelligent synthesized speech speaker verification model includes a speech enhancement network, a feature extraction network, and a feature enhancement module; The training system includes: The acquisition module is used to acquire clean speech datasets and noisy datasets, construct noisy intelligent synthesized speech datasets, and divide them into training sets and test sets; The training module is used during the training phase: The noisy speech in the training set is preprocessed to obtain the corresponding STFT features and Fbank features. The STFT features are used as input to the speech enhancement network. After encoding, the total adaptive fusion features are obtained based on the intermediate transport layer of the speech enhancement network. The total adaptive fusion features are then decoded to obtain the decoded features. The total adaptive fusion feature is used as the input to the feature enhancement module, and the output of the feature enhancement module is concatenated with the Fbank feature to obtain the enhanced feature; The enhanced features are used as input to a feature extraction network to obtain speaker embedding features; The testing module is used during the testing phase to perform model testing based on the test set. The optimization module is used to repeatedly perform training and testing alternately to select the optimal model until the number of training iterations reaches the set maximum value; The speech enhancement network comprises, in sequence, a dense residual convolutional encoder, an adaptive feature fusion transport layer, and a dense residual convolutional decoder. The dense residual convolutional encoder consists of e dilated convolutional units, each of which is composed of a two-dimensional convolutional layer Conv2D, a normalized layer BN, and an activation function PReLU in sequence; and the first e-1 dilated convolutional units are designed as dense residual connections. The adaptive feature fusion transport layer includes a CBAM adaptive feature fusion module and a Sigmoid-gated adaptive feature fusion module; wherein, the CBAM adaptive feature fusion module includes a DPRNN module, a Conformer module and a CBAM module; the Sigmoid-gated adaptive feature fusion module is composed of a two-dimensional convolution Conv2D, a normalization layer BN, and an activation function Sigmoid in sequence. The dense residual convolutional decoder consists of e dilated deconvolutional units, each of which is composed of a two-dimensional deconvolution Conv2DTranspose layer, a normalization layer BN, and an activation function PReLU in sequence; and the first e-1 dilated deconvolutional units are designed with dense residual connections.

8. A storage medium, characterized in that, It stores a computer program for training a robust intelligent synthesized speech speaker verification model, wherein the computer program causes a computer to execute the training method for the robust intelligent synthesized speech speaker verification model as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a training method for performing a robust intelligent synthesized speech speaker verification model as described in any one of claims 1 to 6.

Images