Intelligent synthesized speech speaker verification method based on group-res2block network

Abstract

The application discloses a kind of based on Group-Res2Block network's intelligent synthetic speech speaker confirmation method, comprising:1 obtains intelligent synthetic speech data set;2 establishes the feature extraction network model based on Group-Res2Block;3 trains the feature extraction network model based on Group-Res2Block established;4 realizes prediction using the model established, to reach the purpose of confirming the speaker subject of intelligent synthetic speech.This application can maximize to obtain the common effective voiceprint features of natural human voice and intelligent synthetic speech, so as to effectively realize intelligent synthetic speech speaker confirmation, and can improve confirmation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of speaker recognition, specifically a method for intelligent synthesized speech speaker identification based on a Group-Res2Block network. Background Technology

[0002] With the rise of artificial intelligence technology, intelligent speech synthesis technology can quickly synthesize "a digital voice that sounds like your own," infringing on individuals' right to their voice. This has led to the concept of "voice rights," clarifying that voice, like portrait, has personal attributes. Therefore, when intelligent speech synthesis technology infringes on another person's voice rights, determining the infringing act—that is, identifying the speaker of the intelligently synthesized voice—is a difficult and key issue in current judicial practice.

[0003] In the field of speaker recognition, feature parametric methods and pattern matching were the main early approaches. Feature parametric methods identify different speakers by modeling the human vocal tract; pattern matching methods include Dynamic Time Warping (DTW) and Vector Quantization (VQ). However, both of these methods are highly dependent on manual design. In machine learning methods, traditional Gaussian Mixture Models (GMM) and Gaussian Mixture Universal Background Models (GMM-UBM) can automatically learn the speaker's voiceprint features in a learning manner. However, the recognition accuracy requires a very large amount of sample data.

[0004] With the rise of deep learning, it has demonstrated superior performance in various fields. Neural network methods have applied various networks to the field of speaker recognition, such as Dense Convolutional Networks (DenseNet), Deep Convolutional Networks (VGGVox), Deep Residual Networks (ResNet), Temporal Delay Neural Networks (ECAPA-TDNN) incorporating multi-scale features, RNN networks (BiLSTM), and pure attention networks (Transformer). However, these methods are all based on classic networks, and achieve speaker voiceprint feature extraction by increasing the network depth, simply superimposing the output features of network layers, or using attention mechanisms in the channel and spatial dimensions. They have advantages in feature extraction of natural speech, but their performance on intelligent synthetic speech datasets is not ideal, and there are problems of information redundancy or information loss. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by proposing a method for speaker identification of intelligent synthesized speech based on a Group-Res2Block network. This method aims to maximize the acquisition of common and effective voiceprint features of natural human voice and intelligent synthesized speech, thereby effectively identifying the speaker of intelligent synthesized speech and improving the accuracy of identification.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The present invention provides an intelligent synthesized speech speaker verification method based on a Group-Res2Block network, characterized by the following steps:

[0008] Step 1: Collection of intelligent synthesized speech data and preprocessing of Fbank features:

[0009] Step 1.1: Construct an intelligent synthesized speech dataset:

[0010] Obtain k natural speech samples from H different speakers, and use a speech synthesizer to synthesize the k natural speech samples from H people to obtain k synthesized speech samples from H people, thereby constructing training speech data containing U = 2 × H × k samples;

[0011] Obtain q test groups from N different speakers. Each test group contains one natural speech and one synthesized speech from the same person, thus obtaining V = 2 × q test speech data.

[0012] If the natural speech and synthesized speech in any pair of test groups belong to the same speaker, then the corresponding pair of test groups is designated as a positive sample and labeled as "1"; otherwise, the corresponding pair of test groups is designated as a negative sample and labeled as "0".

[0013] The intelligent synthesized speech dataset consists of U training speech data points and V test speech data points.

[0014] Step 1.2: Use the Fbank feature extractor to perform preliminary feature extraction on the intelligent synthesized speech dataset to obtain the original voiceprint Fbank features;

[0015] Step 2: Establish an intelligent synthesized speech speaker confirmation feature extraction model based on a Group-Res2Block network, which includes: a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module.

[0016] Step 2.1: The feature fusion module under the multi-scale channel attention mechanism consists of a convolutional kernel units of different sizes and a channel attention mechanism module. Each unit is composed of a two-dimensional convolutional Conv2D, a normalization layer BN, and an activation function layer ReLU, and each unit is connected in parallel. The channel attention mechanism module consists of a two-dimensional convolutional Conv2D intermediate layer, another two-dimensional convolutional Conv2D, and a Softmax function, and the intermediate layer is composed of a normalization layer BN and an activation function layer ReLU.

[0017] The original voiceprint Fbank features are input into convolutional kernel units of different sizes for parallel processing. The resulting a features are then concatenated using the concatenate function along their channel dimensions to obtain concatenated total features at different scales, scale_feature. The scale_feature is then input into the channel attention mechanism module, and the Softmax function is used to obtain the weight matrix W of the total features at different scales along the channels. MCA Thus, the fusion feature fusion_feature1 is obtained using equation (1);

[0018] fusion_feature1 = W MCA ×scale_feature (1)

[0019] Step 2.2: The Group-Res2Block module is composed of b Group-Res2Block units in sequence. Each Group-Res2Block unit is composed of one residual unit, a main unit, an activation function layer ReLU, and a normalization layer BN in sequence.

[0020] The main unit and residual unit in the i-th Group-Res2Block unit add the received features, and then process them sequentially through the ReLU activation function layer and the BN normalization layer to obtain the i-th feature mid[i]_feature; the b-th Group-Res2Block unit outputs the b-th feature mid[b]_feature, which is the final output feature of the Group-Res2Block module;

[0021] Step 2.3: The feature fusion module under the layer attention mechanism includes a layer dimension construction module and an attention mechanism module, used to obtain the feature fusion_feature2 under the fusion layer attention mechanism;

[0022] The attention mechanism module consists of an AdaptiveAvgPool layer, an AdaptiveMaxPool layer, a convolution module, and a Softmax function; the fully connected layer fc module consists of two 2D convolutional layers Conv2D and an activation function layer ReLU, wherein the activation function layer ReLU is set between the two 2D convolutional layers Conv2D.

[0023] Step 2.4: The output module includes a data processing module and a fully connected layer, which are used to output speaker embedding features.

[0024] Step 3: Iteratively train the intelligent synthesized speech speaker confirmation feature extraction model by alternating training and testing to obtain the optimal intelligent synthesized speech speaker confirmation feature extraction model.

[0025] The intelligent synthesized speech speaker identification method based on Group-Res2Block network described in this invention is characterized in that each main unit in step 2.2 is composed of a grouping module, two two-dimensional convolutional Conv2D modules, and an SE module;

[0026] The two 2D convolutional Conv2D modules have the same structure, which consists of a convolutional layer, a normalization layer (BN), and an activation function layer (ReLU).

[0027] The SE module sequentially includes: an AdaptiveAvgPool layer, a two-dimensional convolutional Conv2D layer, an activation function layer ReLU, a two-dimensional convolutional Conv2D layer, and a Sigmoid function;

[0028] The received features are input into the first 2D convolutional Conv2D module for processing, and the resulting features are then input into the grouping module.

[0029] For the grouping module, the split function is used to initially divide the features processed by the 2D convolution Conv2D module into c groups. For the first group, the second group is added together to form a new group; for the last group, the second to last group is added together to form a new group; for the middle group, the previous group of the current group is processed by a 3×3 convolution, and then added together with the current group and the next group of the current group to form a new group.

[0030] Perform a 3×3 convolution operation on all new groups and then concatenate them to obtain an intermediate feature M_feature with the same size as fusion_feature1;

[0031] The intermediate feature M_feature is processed sequentially by the second two-dimensional convolutional Conv2D module and the SE module to obtain the feature GR2B_feature of the main unit.

[0032] Step 2.3 includes:

[0033] Step 2.3.1: The layer dimension construction module treats each Group-Res2Block unit as a layer, and uses the i-th feature mid[i]_feature output by the i-th Group-Res2Block unit as the i-th feature Layer[i]_feature of the i-th layer, i = 1, 2, ..., b;

[0034] Expand the layer dimension of the i-th feature Layer[i]_feature to obtain the i-th layer feature after expansion, which includes the layer dimension, channel dimension, width dimension and height dimension; where the layer dimension is set to "1"; merge the i-th layer feature after expansion in its width dimension and height dimension to obtain the merged i-th layer feature new_Layer[i]_feature.

[0035] The layer dimension construction module uses the concatnate function to concatenate the merged b-layer features {new_Layer[i]_feature|i=1,2,......,b} in the layer dimension to obtain the concatenated feature all_Layer_feature;

[0036] Step 2.3.2: Input the concatenated features (all_Layer_feature) into the attention mechanism module. After processing by the AdaptiveAvgPool layer and the AdaptiveMaxPool layer, they are then input into the convolution module for further processing. The two resulting features are added together, and then the Softmax function is used to process the added features to obtain the weight matrix W in the layer dimension. LA Using the weight matrix W in equation (2) LA Multiplying the concatenated feature all_Layer_feature by the feature itself yields the feature fusion_feature2 under the attention mechanism of the fusion layer;

[0037] fusion_feature2 = W LA ×all_Layer_feature (2).

[0038] Step 2.4 includes:

[0039] Step 2.4.1: The data processing module performs a sum operation on fusion_feature2 in the layer dimension to restore the layer dimension to 1, and then removes the layer dimension to obtain a dimensionality-reduced feature that only contains the channel dimension, width dimension, and height dimension. Then, the dimensionality-reduced feature is processed by Self-Attention Pooling to obtain sentence-level features.

[0040] Step 2.4.2: After the fully connected layer reduces the dimension of the obtained sentence-level features to C dimensions, the speaker embedding features output by the intelligent synthesized speech speaker confirmation feature extraction model are obtained.

[0041] Step 3 includes:

[0042] Step 3.1: Construct the category loss function L using equation (3).AMSoftmax :

[0043]

[0044] In equation (3), U represents the total number of speech data in the intelligent synthesized speech dataset, m and s are hyperparameters, and f k y represents the speaker embedding feature output by the intelligent synthesized speech speaker confirmation feature extraction model for the k-th training speech sample. k ={1,2,3...H} represents the speaker embedding feature f output by the intelligent synthesized speech speaker confirmation feature extraction model after processing any k-th training speech sample in the training speech data. k The speaker's category The y-th element of the model network weight matrix W represents the weight matrix W of the network. k Column parameters, T represents transpose, W j This represents the parameter in the j-th column of the weight matrix W, where j = {1, 2, 3, ..., H};

[0045] Step 3.2: Optimize the network parameters of the intelligent synthesized speech speaker confirmation feature extraction model using the Adam optimizer, and adjust the category loss function L... AMSoftmax The training iteration count is gradually reduced. When the number of training iterations e is an integer multiple of the set test interval g, the current model and its parameters are saved, and the testing phase of step 3.2 is entered.

[0046] Step 3.3, the testing phase includes: the scoring process and the threshold determination process;

[0047] Step 3.3.1: During the scoring process, input q pairs of test groups into the saved current model to obtain the speaker embedding features of each pair of test groups; use the cosine distance formula to score the speaker embedding features of each pair of test groups using cosine distance.

[0048] Step 3.3.2, the threshold determination process involves iterating over the threshold from its minimum value to its maximum value within a set threshold range. During each iteration, the score of q for the test group is compared with the threshold value of each iteration.

[0049] If the score of a test group is greater than the threshold of the current iteration, the corresponding test group is determined to be a positive sample and the label of the corresponding test group is set to "1"; otherwise, the corresponding test group is determined to be a negative sample and the label of the corresponding test group is set to "0".

[0050] The judgment result of the test group for each q pair in each iteration is compared with the actual label of the corresponding test group to calculate the equal error rate (EER) value for each iteration and complete one test phase.

[0051] Step 3.4: Train again according to the process in Step 3.2. When the number of training iterations e reaches an integer multiple of the set test interval g, test again according to the process in Step 3.3. In this way, the model is processed by alternating training and testing until the number of training iterations reaches the set maximum value. The model with the smallest error rate EER value in all testing stages is determined as the optimal intelligent synthesized speech speaker confirmation feature extraction model.

[0052] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the intelligent synthesized speech speaker verification methods, and the processor is configured to execute the program stored in the memory.

[0053] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs any of the steps of the intelligent synthesized speech speaker confirmation method.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] 1. This invention improves upon Res2Block by proposing Group-Res2Block, which merges the original groups to generate new groups, strengthening the contextual relationship of feature information. This allows for effective modeling of the voiceprint features of intelligent synthesized speech using this local correlation, which helps improve the accuracy of speaker identification.

[0056] 2. This invention utilizes convolutional kernels of different sizes in parallel at the front end of the main Res2Block module to generate multi-scale features, and introduces a channel attention mechanism into the multi-scale features to effectively capture as many input Fbank features as possible. This expands the receptive field of feature information, making the voiceprint feature information entering the main network richer, and also effectively selects channel features, avoiding redundancy of voiceprint feature information in the channel dimension.

[0057] 3. This invention creates layer dimensions and introduces an attention mechanism into the network layers, enabling features from deep and shallow layers to participate in the final feature expression to different degrees. This allows for effective selection of deep and shallow features, resulting in a more complete final feature expression and accelerating the convergence speed of the model to a certain extent, thereby improving the recognition speed of speech speaker confirmation. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0059] Figure 2 This is a diagram of the Fbank feature extractor of the present invention;

[0060] Figure 3 This is a structural diagram of the channel attention mechanism module of the present invention;

[0061] Figure 4 This is a structural diagram of the feature fusion module under the multi-scale channel attention mechanism of the present invention;

[0062] Figure 5 This is a structural diagram of the Group-Res2Block of the present invention;

[0063] Figure 6 This is a structural diagram of the feature fusion module under the multi-scale layer attention mechanism of the present invention;

[0064] Figure 7 This is a structural diagram of the attention mechanism module introduced in the layer attention mechanism of this invention. Detailed Implementation

[0065] In this embodiment, an intelligent synthesized speech speaker verification method based on a Group-Res2Block network mainly utilizes the fusion of Group-Res2Block, multi-scale features, and attention mechanisms to effectively extract intelligent synthesized speech features and natural speech features, such as... Figure 1 As shown, the specific steps are as follows:

[0066] Step 1: Collection of intelligent synthesized speech data and preprocessing of Fbank features:

[0067] Step 1.1: Construct an intelligent synthesized speech dataset:

[0068] Obtain k natural speech samples from T different speakers, and use a speech synthesizer to synthesize the k natural speech samples from T speakers to obtain k synthesized speech samples from T speakers, thereby constructing training speech data containing U = 2 × T × k samples;

[0069] Obtain q test groups from N different speakers. Each test group contains one natural speech and one synthesized speech from the same person, thus obtaining V = 2 × q test speech data.

[0070] If the natural speech and synthesized speech in any pair of test groups belong to the same speaker, then the corresponding pair of test groups is designated as a positive sample and labeled as "1"; otherwise, the corresponding pair of test groups is designated as a negative sample and labeled as "0".

[0071] The intelligent synthesized speech dataset consists of U training speech data points and V test speech data points.

[0072] In this embodiment, the natural speech is from the Chinese open-source dataset aishell3, and the intelligent synthesized speech is synthesized from the aishell3 dataset using the FastSpeech2 synthesizer. This is because the FastSpeech2 algorithm produces high-quality synthesized speech and has a fast synthesis speed, making it representative of intelligent synthesized speech. The training data samples mix natural and synthesized speech, containing 130 different speakers, with 200 natural and 200 synthesized speech samples from each speaker. The test data samples contain 40 different speakers, input in the form of data sets, with each set consisting of one natural speech sample and one synthesized speech sample, for a total of 11,860 test sets. Since the optimization goal during training is to improve the accuracy of speaker classification, the test results are obtained by scoring the cosine distance between the speaker embedding features of two speech samples and comparing it with a threshold. Therefore, different data sample input formats are used. The training and test data samples are first processed using VAD technology to remove silent segments, and then the speech length is uniformly truncated to 2 seconds. VAD technology ensures that all speech sent to the network contains speech information, and the uniform audio length is used to accelerate the training process using batch processing. The 2-second audio already contains sufficient speaker voiceprint features.

[0073] Step 1.2, using, for example Figure 2 The Fbank feature extractor shown performs preliminary feature extraction on the intelligent synthesized speech dataset to obtain the original voiceprint Fbank features; the Fbank features can fully preserve the information of the original input speech.

[0074] Step 2: Establish an intelligent synthesized speech speaker confirmation feature extraction model based on a Group-Res2Block network, which includes: a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module.

[0075] Step 2.1: The feature fusion module under the multi-scale channel attention mechanism consists of *a* convolutional kernel units of different sizes and a channel attention mechanism module. Each unit is composed of a 2D convolutional layer (Conv2D), a normalization layer (BN), and an activation function layer (ReLU) in sequence, and each unit is connected in parallel; for example... Figure 3 As shown, the channel attention mechanism module consists of a two-dimensional convolutional Conv2D intermediate layer, another two-dimensional convolutional Conv2D layer, and a Softmax function in sequence. The intermediate layer is composed of a normalization layer BN and an activation function layer ReLU in sequence.

[0076] like Figure 4As shown, the original voiceprint Fbank features are input into convolutional kernel units of different sizes for parallel processing. The resulting a features are then concatenated using the concatenate function along their channel dimensions to obtain the concatenated total features at different scales, scale_feature. The scale_feature is then input into the channel attention mechanism module, and the Softmax function is used to obtain the weight matrix W of the total features at different scales along the channels. MCA Thus, the fusion feature fusion_feature1 is obtained using equation (1). The features extracted by different convolutional kernels will have certain differences. At the same time, the parallel structure can obtain feature information of different receptive fields of the same level. The fused features are then passed to the next layer to supplement the features extracted by different convolutional kernels. Since each channel of a feature can express different features, each channel has the same weight when no processing is performed, that is, it has the same contribution to the later processing. However, in reality, each channel is not equivalent to the final expression of the feature. Channel attention (CA) is used to assign corresponding weights to each channel, that is, to obtain the importance of different channels. In this way, features with high weights in the total different scale channel dimensions can be extracted, reducing unnecessary redundant features.

[0077] fusion_feature1 = W MCA ×scale_feature (1)

[0078] Step 2.2: The Group-Res2Block module is composed of b Group-Res2Block units, as follows: Figure 5 As shown, each Group-Res2Block unit consists of a residual unit, a main unit, an activation function layer (ReLU), and a normalization layer (BN) in sequence.

[0079] The main unit and residual unit in the i-th Group-Res2Block unit add the received features, and then process them sequentially through the ReLU activation function layer and the BN normalization layer to obtain the i-th feature mid[i]_feature; the b-th Group-Res2Block unit outputs the b-th feature mid[b]_feature, which is the final output feature of the Group-Res2Block module;

[0080] In this embodiment, each main unit consists of a grouping module, two two-dimensional convolutional Conv2D modules, and an SE module;

[0081] The two 2D convolutional Conv2D modules have the same structure, consisting of a convolutional layer, a normalization layer (BN), and an activation function layer (ReLU) in sequence.

[0082] The SE module consists of an AdaptiveAvgPool layer, a 2D Conv2D convolutional layer, a ReLU activation function layer, a 2D Conv2D convolutional layer, and a Sigmoid function.

[0083] The received features are input into the first 2D convolutional Conv2D module, and then the resulting features are input into the grouping module.

[0084] For the grouping module, the split function is used to initially divide the features processed by the 2D convolutional Conv2D module into p groups. For the first group, the second group is added together to form a new group; for the last group, the second to last group is added together to form a new group; for the middle group, the previous group of the current group is convolved with 3×3, and then added together with the current group and the next group of the current group to form a new group. Since the new group contains the feature information of the current group, the previous group, and the next group, compared with the original Res2Block, it can not only obtain multi-scale feature representation, but also connect with the context and enhance the relevance of speech feature information.

[0085] Perform a 3×3 convolution operation on all new groups and then concatenate them to obtain an intermediate feature M_feature with the same size as fusion_feature1;

[0086] The intermediate feature M_feature is processed sequentially by the second 2D convolutional Conv2D module and the SE module to obtain the feature GR2B_feature of the main unit.

[0087] Step 2.3, as follows Figure 6 As shown, the feature fusion module under the multi-scale layer attention mechanism includes: a layer dimension construction module and an attention mechanism module; as Figure 7 As shown, the attention mechanism module consists of an AdaptiveAvgPool layer, an AdaptiveMaxPool layer, a convolution module, and a Softmax function; the fully connected layer fc module consists of two 2D convolutional Conv2D layers and an activation function layer ReLU, wherein the activation function layer ReLU is placed between the two 2D convolutional Conv2D layers.

[0088] Step 2.3.1: The layer dimension construction module treats each Group-Res2Block unit as a layer, and uses the i-th feature mid[i]_feature output by the i-th Group-Res2Block unit as the i-th feature Layer[i]_feature of the i-th layer, i = 1, 2, ..., b;

[0089] Expand the layer dimension of the i-th feature Layer[i]_feature to obtain the expanded layer i-th feature containing the layer dimension, channel dimension, width dimension, and height dimension; where the layer dimension is set to "1"; merge the expanded layer i-th feature in its width and height dimensions to obtain the merged layer i-th feature new_Layer[i]_feature; the layer dimension is established to ensure that the weights obtained by the attention mechanism act on the complete feature output of each layer;

[0090] The layer dimension construction module uses the concatnate function to concatenate the merged b-layer features {new_Layer[i]_feature|i=1,2,......,b} in the layer dimension to obtain the concatenated feature all_Layer_feature;

[0091] Step 2.3.2: Input the concatenated features (all_Layer_feature) into the attention mechanism module. After processing by the AdaptiveAvgPool layer and the AdaptiveMaxPool layer, they are then input into the convolution module for further processing. The resulting two features are added together, and then the Softmax function is used to process the added features to obtain the weight matrix W in the layer dimension. LA Using the weight matrix W in equation (2) LA Multiplying the concatenated features `all_Layer_feature` yields the features `fusion_feature2` under the attention mechanism of the fusion layer. Introducing an attention mechanism assigns different weights to each layer, ensuring each layer participates to varying degrees in the final feature representation, resulting in multi-scale optimal fusion features from both shallow and deep layers. Furthermore, since the final output features contain features from different layers, the feature information is richer and more complete, which can accelerate network convergence to some extent. The weight matrix W is obtained using the Softmax function. LA The sum of all values ​​in the layer equals 1, thus normalizing the contribution of each layer to the final feature representation;

[0092] fusion_feature2 = W LA ×all_Layer_feature (2)

[0093] Step 2.4: The output module consists of a data processing module and a fully connected layer.

[0094] Step 2.4.1: The data processing module performs a sum operation on the layer dimension of fusion_feature2 to restore the layer dimension to 1, and then removes the layer dimension to obtain a dimensionality-reduced feature that only contains the channel dimension, width dimension, and height dimension. Then, the dimensionality-reduced feature is processed by Self-Attention Pooling to obtain sentence-level features. Self-Attention Pooling is a pooling method with a self-attention mechanism that can perform frame-by-frame attention weighting on the dimensionality-reduced features to obtain sentence-level features that contain the most speaker voiceprint information.

[0095] Step 2.4.2: After the fully connected layer reduces the dimension of the obtained sentence-level features to C dimensions, the speaker embedding features output by the intelligent synthesized speech speaker confirmation feature extraction model are obtained.

[0096] Step 3: Iteratively train the intelligent synthesized speech speaker confirmation feature extraction model:

[0097] Step 3.1: Construct the category loss L using equation (3). AMSoftmax :

[0098]

[0099] In equation (3), U represents the total number of speech data in the intelligent synthesized speech dataset, m and s are hyperparameters, and f k y represents the speaker embedding feature output by the intelligent synthesized speech speaker confirmation feature extraction model for the k-th training speech sample. k ={1,2,3...H} represents the speaker embedding feature f output by the intelligent synthesized speech speaker confirmation feature extraction model after processing any k-th training speech sample in the training speech data. k The speaker's category The y-th element of the model network weight matrix W represents the weight matrix W of the network. k Column parameters, W j The j-th column parameter of the weight matrix W is represented by j = {1, 2, 3, ..., H}; T represents the transpose; the loss function L AMSoftmax By optimizing the angle θ to reduce intra-class differences and increase inter-class differences, classification accuracy is improved. In this embodiment, s is set to 30 and m is set to 0.1.

[0100] Step 3.2: Optimize the network parameters of the intelligent synthesized speech speaker confirmation feature extraction model using the Adam optimizer, and adjust the category loss function L... AMSoftmaxThe training iteration count is gradually reduced. When the number of training iterations e is an integer multiple of the set test interval g, the current model and its parameters are saved, and the test phase in step 3.2 is entered. In this embodiment, 10 speakers are randomly selected from the training data samples in each iteration, and 5 natural speech and 5 intelligent synthesized speech are randomly selected from each person's data. Therefore, the batch size in each iteration is 100. The maximum number of training iterations e is set to 500, and the test interval g is set to 10. The training process adopts a preheating training strategy, which helps to alleviate the overfitting phenomenon of the network and improve the accuracy. The specific strategy is as follows: the initial learning rate is set to 0.001, and the learning rate is reduced to 0.95 times the original after each test. The training ends after 500 iterations.

[0101] Step 3.3, the testing phase includes: the scoring process and the threshold determination process;

[0102] Step 3.3.1: During the scoring process, input q pairs of test groups into the saved current model to obtain the speaker embedding features of each pair of test groups; use the cosine distance formula to score the speaker embedding features of each pair of test groups using cosine distance.

[0103] Step 3.3.2, the threshold determination process involves iterating the threshold from the minimum to the maximum value within the set threshold range with a step size r. In each iteration, the score of the test group by q is compared with the threshold of each iteration. In this embodiment, the threshold range is determined by the maximum and minimum values ​​of the cosine distance score of the test group in this test. The equal error rate (EER) value of the total test sample data is calculated starting from the minimum value, and the step size r of each iteration is 0.01.

[0104] If the score of a test group is greater than the threshold of the current iteration, the test group is determined to be a positive sample and the label of the corresponding test group is set to "1"; otherwise, the test group is determined to be a negative sample and the label of the corresponding test group is set to "0".

[0105] The judgment result of the q-pair test group in each iteration is compared with the actual label of the corresponding test group to calculate the equal error rate (EER) value of each iteration and complete one test phase.

[0106] Step 3.4: Train again according to the process in Step 3.2. When the number of training iterations e reaches an integer multiple of the set test interval g, test again according to the process in Step 3.3. In this way, the model is processed by alternating training and testing until the number of training iterations reaches the set maximum value. The model with the smallest error rate EER value in all testing stages is determined as the optimal intelligent synthesized speech speaker confirmation feature extraction model.

[0107] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0108] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A method for intelligent synthesized speech speaker verification based on a Group-Res2Block network, characterized in that, The procedure is as follows: Step 1: Collection of intelligent synthesized speech data and preprocessing of Fbank features: Step 1.1: Construct an intelligent synthesized speech dataset: Obtain k natural speech samples from H different speakers, and use a speech synthesizer to synthesize the k natural speech samples from H people to obtain k synthesized speech samples from H people, thereby constructing training speech data containing U=2×H×k samples; Obtain q test groups from N different speakers. Each test group contains one natural speech and one synthesized speech from the same person, thus obtaining V = 2 × q test speech data. If the natural speech and synthesized speech in any pair of test groups belong to the same speaker, then the corresponding pair of test groups is designated as a positive sample and labeled "1"; otherwise, the corresponding pair of test groups is designated as a negative sample and labeled "0". The intelligent synthesized speech dataset consists of U training speech data points and V test speech data points. Step 1.2: Use the Fbank feature extractor to perform preliminary feature extraction on the intelligent synthesized speech dataset to obtain the original voiceprint Fbank features; Step 2: Establish an intelligent synthesized speech speaker confirmation feature extraction model based on a Group-Res2Block network, which includes: a feature fusion module under a multi-scale channel attention mechanism, a Group-Res2Block module, a feature fusion module under a layer attention mechanism, and an output module. Step 2.1: The feature fusion module under the multi-scale channel attention mechanism consists of a convolutional kernel units of different sizes and a channel attention mechanism module. Each unit is composed of a two-dimensional convolutional Conv2D, a normalization layer BN, and an activation function layer ReLU, and each unit is connected in parallel. The channel attention mechanism module consists of a two-dimensional convolutional Conv2D intermediate layer, another two-dimensional convolutional Conv2D, and a Softmax function, and the intermediate layer is composed of a normalization layer BN and an activation function layer ReLU. The original voiceprint Fbank features are input into convolutional kernel units of different sizes for parallel processing. The resulting a features are then concatenated using the concatenate function along their channel dimensions to obtain concatenated total features at different scales, scale_feature. The scale_feature is then input into the channel attention mechanism module, and the Softmax function is used to obtain the weight matrix of the total features at different scales along the channels. Thus, the fusion feature fusion_feature1 is obtained using equation (1); (1) Step 2.2: The Group-Res2Block module is composed of b Group-Res2Block units in sequence. Each Group-Res2Block unit is composed of one residual unit, a main unit, an activation function layer ReLU, and a normalization layer BN in sequence. The main unit and residual unit in the i-th Group-Res2Block unit add the received features, and then process them sequentially through the ReLU activation function layer and the BN normalization layer to obtain the i-th feature mid[i]_feature; the b-th Group-Res2Block unit outputs the b-th feature mid[b]_feature, which is the final output feature of the Group-Res2Block module; Each main unit in step 2.2 consists of a grouping module, two 2D convolutional Conv2D modules, and an SE module; The two 2D convolutional Conv2D modules have the same structure, which consists of a convolutional layer, a normalization layer (BN), and an activation function layer (ReLU). The SE module sequentially includes: an AdaptiveAvgPool layer, a two-dimensional convolutional Conv2D layer, an activation function layer ReLU, a two-dimensional convolutional Conv2D layer, and a Sigmoid function; The received features are input into the first 2D convolutional Conv2D module for processing, and the resulting features are then input into the grouping module. For the grouping module, the split function is used to initially divide the features processed by the 2D convolution Conv2D module into c groups. For the first group, the second group is added together to form a new group; for the last group, the second to last group is added together to form a new group; for the middle group, the previous group of the current group is processed by a 3×3 convolution, and then added together with the current group and the next group of the current group to form a new group. Perform a 3×3 convolution operation on all new groups and then concatenate them to obtain an intermediate feature M_feature with the same size as fusion_feature1; The intermediate feature M_feature is processed sequentially by the second two-dimensional convolutional Conv2D module and the SE module to obtain the feature GR2B_feature of the main unit. Step 2.3: The feature fusion module under the layer attention mechanism includes a layer dimension construction module and an attention mechanism module, used to obtain the feature fusion_feature2 under the fusion layer attention mechanism; The attention mechanism module consists of an AdaptiveAvgPool layer, an AdaptiveMaxPool layer, a convolution module, and a Softmax function; the fully connected layer fc module consists of two 2D convolutional layers Conv2D and an activation function layer ReLU, wherein the activation function layer ReLU is set between the two 2D convolutional layers Conv2D. Step 2.4: The output module includes a data processing module and a fully connected layer, which are used to output speaker embedding features. Step 3: Iteratively train the intelligent synthesized speech speaker confirmation feature extraction model by alternating training and testing to obtain the optimal intelligent synthesized speech speaker confirmation feature extraction model.

2. The intelligent synthesized speech speaker verification method based on Group-Res2Block network according to claim 1, characterized in that, Step 2.3 includes: Step 2.3.1: The layer dimension construction module treats each Group-Res2Block unit as a layer, and uses the i-th feature mid[i]_feature output by the i-th Group-Res2Block unit as the i-th feature Layer[i]_feature of the i-th layer, i=1,2,......,b; Expand the layer dimension of the i-th feature Layer[i]_feature to obtain the i-th layer feature after expansion, which includes the layer dimension, channel dimension, width dimension and height dimension; where the layer dimension is set to "1"; merge the i-th layer feature after expansion in its width dimension and height dimension to obtain the merged i-th layer feature new_Layer[i]_feature. The layer dimension construction module uses the concatnate function to concatenate the merged b-layer features {new_Layer[i]_feature|i=1,2,......,b} in the layer dimension to obtain the concatenated feature all_Layer_feature; Step 2.3.2: Input the concatenated features (all_Layer_feature) into the attention mechanism module. After processing by the AdaptiveAvgPool layer and the AdaptiveMaxPool layer, they are then input into the convolution module for further processing. The two resulting features are added together, and then the Softmax function is used to process the added features to obtain the weight matrix in the layer dimension. Using the weight matrix of equation (2) Multiplying the concatenated feature all_Layer_feature by the feature itself yields the feature fusion_feature2 under the attention mechanism of the fusion layer; (2)。 3. The intelligent synthesized speech speaker verification method based on Group-Res2Block network according to claim 2, characterized in that, Step 2.4 includes: Step 2.4.1: The data processing module performs a sum operation on fusion_feature2 in the layer dimension to restore the layer dimension to 1, and then removes the layer dimension to obtain a dimensionality-reduced feature that only contains the channel dimension, width dimension, and height dimension. Then, the dimensionality-reduced feature is processed by Self-Attention Pooling to obtain sentence-level features. Step 2.4.2: After the fully connected layer reduces the dimension of the obtained sentence-level features to C dimensions, the speaker embedding features output by the intelligent synthesized speech speaker confirmation feature extraction model are obtained.

4. The intelligent synthesized speech speaker verification method based on Group-Res2Block network according to claim 3, characterized in that, Step 3 includes: Step 3.1: Construct the category loss function using equation (3). : (3) In equation (3), U represents the total number of speech data in the intelligent synthesized speech dataset, and m and s are hyperparameters. This represents the speaker embedding feature output by the intelligent synthesized speech speaker confirmation feature extraction model after processing the k-th training speech sample. This represents the speaker embedding feature output by the intelligent synthesized speech speaker confirmation feature extraction model after processing any k-th training speech sample in the training speech data. The speaker's category Represents the network weight matrix of the model The Column parameters, T represents transpose, Represents the weight matrix The j-th column parameter, ; Step 3.2: Optimize the network parameters of the intelligent synthesized speech speaker confirmation feature extraction model using the Adam optimizer, and adjust the category loss function. The training iteration count is gradually reduced. When the number of training iterations e is an integer multiple of the set test interval g, the current model and its parameters are saved, and the testing phase of step 3.2 is entered. Step 3.3, the testing phase includes: the scoring process and the threshold determination process; Step 3.3.1: During the scoring process, input q pairs of test groups into the saved current model to obtain the speaker embedding features of each pair of test groups; use the cosine distance formula to score the speaker embedding features of each pair of test groups using cosine distance. Step 3.3.2, the threshold determination process involves iterating over the threshold from its minimum value to its maximum value within a set threshold range. During each iteration, the score of q for the test group is compared with the threshold value of each iteration. If the score of a test group is greater than the threshold of the current iteration, the corresponding test group is determined to be a positive sample and the label of the corresponding test group is set to "1"; otherwise, the corresponding test group is determined to be a negative sample and the label of the corresponding test group is set to "0". The judgment result of the test group for each q pair in each iteration is compared with the actual label of the corresponding test group to calculate the equal error rate (EER) value for each iteration and complete one test phase. Step 3.4: Train again according to the process in Step 3.

2. When the number of training iterations e reaches an integer multiple of the set test interval g, test again according to the process in Step 3.

3. In this way, the model is processed by alternating training and testing until the number of training iterations reaches the set maximum value. The model with the smallest error rate EER value in all testing stages is determined as the optimal intelligent synthesized speech speaker confirmation feature extraction model.

5. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the intelligent synthesized speech speaker verification method according to any one of claims 1-4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the intelligent synthesized speech speaker verification method according to any one of claims 1-4.

Images