Speaker verification model training method, device and equipment
By inputting speech features into the global and local feature extraction network in the speaker confirmation model and adding a data processing network between the TDNN and the Transformer network, the problem that the network cannot learn in parallel in the prior art is solved, and the model performance is improved.
Patent Information
- Application Number
- CN202211679705.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-26
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2042-12-26
AI Technical Summary
In the prior art, the TDNN network and the Transformer network cannot be learned in parallel in serial, resulting in poor performance of the speaker confirmation model.
By inputting speech features into the global feature extraction network and local feature extraction network in the initial speaker confirmation model, the target global feature and target local feature are obtained, and a data processing network is added between the TDNN and the Transformer network, so that the two can learn in parallel.
The parallel learning of TDNN and Transformer networks is realized, which improves the performance of the speaker confirmation model and reduces the error rate of the same.
Smart Images
Figure CN116129885B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text-independent speaker verification, and in particular to a speaker verification model training method, device and equipment. Background Art
[0002] Currently, in text-independent speaker verification technology, it is usually necessary to train an initial speaker verification model to obtain a speaker verification model. The speaker verification model is used, for example, to extract feature vectors of two speech data and determine whether the two speech data come from the same speaker based on the extracted feature vectors.
[0003] In the related art, the initial speaker confirmation model includes a serial time delay neural network (TDNN) and a transformer (Transformer) network. During the training process of the initial speaker confirmation model, since the TDNN network and the Transformer network are serial, the TDNN network and the Transformer network cannot be learned in parallel, resulting in poor performance of the speaker confirmation model. Summary of the invention
[0004] The present invention provides a speaker verification model training method to solve the defects in the prior art that a TDNN network and a Transformer network cannot be learned in parallel and the performance of the speaker verification model is poor, thereby achieving the purpose of improving the performance of the speaker verification model.
[0005] In a first aspect, the present invention provides a speaker verification model training method, comprising:
[0006] Obtaining speech features corresponding to all speech samples in a training batch and label information corresponding to all speech samples;
[0007] Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain target global features and target local features corresponding to the speech features;
[0008] Determining prediction information corresponding to all speech samples based on the target global feature and the target local feature;
[0009] The model parameters of the initial speaker verification model are updated according to the label information and the prediction information to obtain a speaker verification model.
[0010] According to a speaker verification model training method provided by the present invention, the speech features are respectively input into a global feature extraction network and a local feature extraction network in an initial speaker verification model to obtain target global features and target local features corresponding to the speech features, including:
[0011] Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain a first global feature and a first local feature;
[0012] Inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature;
[0013] Inputting the second global feature into the local feature extraction network to obtain a second local feature and the target local feature;
[0014] Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions;
[0015] The third local feature is input into the global feature extraction network to obtain a target global feature, and the dimension of the third local feature is the same as that of the first global feature.
[0016] According to a speaker verification model training method provided by the present invention, the step of inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature includes:
[0017] Performing upsampling processing on the first global feature through the first data processing network to obtain an upsampled feature, wherein the upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension;
[0018] Performing convolution processing on the upsampled feature through the first data processing network to obtain a first convolution feature, wherein the first convolution feature and the upsampled feature have the same audio frame number dimension and a different audio frequency dimension;
[0019] Performing batch normalization processing on the first convolutional features through the first data processing network to obtain first normalized features;
[0020] The product of the first normalized feature and a first preset adjustable parameter is calculated through the first data processing network to obtain the second global feature.
[0021] According to a speaker verification model training method provided by the present invention, the step of inputting the second local feature into the second data processing network in the initial speaker verification model to obtain the third local feature includes:
[0022] The second local feature is convolved by the second data processing network to obtain a second convolution feature; the second local feature and the second convolution feature have different audio frequency dimensions and different audio frame number dimensions;
[0023] Performing layer normalization processing on the second convolutional features through the second data processing network to obtain second normalized features;
[0024] The product of the second normalized feature and a second preset adjustable parameter is calculated through the second data processing network to obtain the third local feature.
[0025] According to a speaker verification model training method provided by the present invention, the step of obtaining speech features corresponding to all speech samples in a training batch includes:
[0026] Extracting Mel-frequency cepstral coefficient features of all speech samples;
[0027] The Mel frequency cepstral coefficient feature is input into a one-dimensional convolutional layer in the initial speaker verification model to obtain the speech feature.
[0028] According to a speaker verification model training method provided by the present invention, the step of determining prediction information corresponding to all speech samples based on the target global features and the target local features includes:
[0029] Superimposing the target global feature and the target local feature to obtain a fused feature;
[0030] Inputting the fused features into the attention statistics pooling layer in the initial speaker verification model;
[0031] Inputting the output result of the attention statistics pooling layer into the first batch of normalization layers in the initial speaker verification model to obtain a third normalized feature;
[0032] Inputting the third normalized feature into a fully connected layer in the initial speaker verification model;
[0033] Inputting the output result of the fully connected layer into the second batch normalization layer in the initial speaker verification model to obtain fourth normalized features;
[0034] The fourth normalized feature is input into a classifier in the initial speaker verification model to obtain the prediction information.
[0035] According to a speaker verification model training method provided by the present invention, the model parameters of the initial speaker verification model are updated according to the label information and the prediction information to obtain the speaker verification model, including:
[0036] Calculating a smoothed cross entropy loss value between the label information and the prediction information;
[0037] Based on the smoothed cross entropy loss value, model parameters of the initial speaker verification model are updated to obtain a speaker verification model.
[0038] In a second aspect, the present invention further provides a speaker verification model training device, comprising:
[0039] An acquisition module, used to acquire speech features corresponding to all speech samples in a training batch, and label information corresponding to all speech samples;
[0040] A feature extraction module, used to input the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model, respectively, to obtain target global features and target local features corresponding to the speech features;
[0041] A determination module, configured to determine prediction information corresponding to all speech samples based on the target global feature and the target local feature;
[0042] An updating module is used to update the model parameters of the initial speaker confirmation model according to the label information and the prediction information to obtain a speaker confirmation model.
[0043] According to a speaker verification model training device provided by the present invention, the feature extraction module is specifically used for:
[0044] Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain a first global feature and a first local feature;
[0045] Inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature;
[0046] Inputting the second global feature into the local feature extraction network to obtain a second local feature and the target local feature;
[0047] Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions;
[0048] The third local feature is input into the global feature extraction network to obtain a target global feature, and the dimension of the third local feature is the same as that of the first global feature.
[0049] According to a speaker verification model training device provided by the present invention, the feature extraction module is specifically used for:
[0050] Performing upsampling processing on the first global feature through the first data processing network to obtain an upsampled feature, wherein the upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension;
[0051] Performing convolution processing on the upsampled feature through the first data processing network to obtain a first convolution feature, wherein the first convolution feature and the upsampled feature have the same audio frame number dimension and a different audio frequency dimension;
[0052] Performing batch normalization processing on the first convolutional features through the first data processing network to obtain first normalized features;
[0053] The product of the first normalized feature and a first preset adjustable parameter is calculated through the first data processing network to obtain the second global feature.
[0054] According to a speaker verification model training device provided by the present invention, the feature extraction module is specifically used for:
[0055] The second local feature is convolved by the second data processing network to obtain a second convolution feature; the second local feature and the second convolution feature have different audio frequency dimensions and different audio frame number dimensions;
[0056] Performing layer normalization processing on the second convolutional features through the second data processing network to obtain second normalized features;
[0057] The product of the second normalized feature and a second preset adjustable parameter is calculated through the second data processing network to obtain the third local feature.
[0058] According to a speaker verification model training device provided by the present invention, the acquisition module is specifically used for:
[0059] Extracting Mel-frequency cepstral coefficient features of all speech samples;
[0060] The Mel frequency cepstral coefficient feature is input into a one-dimensional convolutional layer in the initial speaker verification model to obtain the speech feature.
[0061] According to a speaker verification model training device provided by the present invention, the determination module is specifically used for:
[0062] Superimposing the target global feature and the target local feature to obtain a fused feature;
[0063] Inputting the fused features into the attention statistics pooling layer in the initial speaker verification model;
[0064] Inputting the output result of the attention statistics pooling layer into the first batch of normalization layers in the initial speaker verification model to obtain a third normalized feature;
[0065] Inputting the third normalized feature into a fully connected layer in the initial speaker verification model;
[0066] Inputting the output result of the fully connected layer into the second batch normalization layer in the initial speaker verification model to obtain fourth normalized features;
[0067] The fourth normalized feature is input into a classifier in the initial speaker verification model to obtain the prediction information.
[0068] According to a speaker verification model training device provided by the present invention, the updating module is specifically used for:
[0069] Calculating a smoothed cross entropy loss value between the label information and the prediction information;
[0070] Based on the smoothed cross entropy loss value, model parameters of the initial speaker verification model are updated to obtain a speaker verification model.
[0071] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speaker confirmation model training method described above is implemented.
[0072] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described speaker verification model training methods.
[0073] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speaker verification model training method described in any one of the above is implemented.
[0074] The present invention provides a speaker verification model training method, device and equipment, which respectively input speech features into the global feature extraction network and the local feature extraction network in the initial speaker verification model, determine the prediction information corresponding to all speech samples, and update the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain the speaker verification model, thereby solving the defect in the related art that the TDNN network and the Transformer network cannot be learned in parallel, and improving the performance of the speaker verification model. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0076] Figure 1 It is a flowchart of the speaker verification model training method provided by the present invention;
[0077] Figure 2 is a schematic diagram of an initial speaker confirmation model provided by the present invention;
[0078] Figure 3 is a schematic diagram of the structure of a first data processing network provided by the present invention;
[0079] Figure 4 is a schematic diagram of the structure of a second data processing network provided by the present invention;
[0080] Figure 5 It is a structural schematic diagram of the speaker confirmation model training device provided by the present invention;
[0081] Figure 6 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0082] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0083] In the prior art, the initial speaker verification model includes a serial time delay neural network (TDNN) and a transformer (Transformer) network. During the training of the initial speaker verification model, since the TDNN and the transformer network are serial, the TDNN and the transformer network cannot be learned in parallel, resulting in that the learning results of the TDNN and the transformer network cannot complement each other, resulting in poor performance of the speaker verification model and a high error rate.
[0084] In the present invention, in order to improve the performance of the speaker verification model and reduce the equal error rate, the inventors connect the TDNN and Transformer networks in parallel and add a data processing network between the TDNN and Transformer networks so that the learning results of the two can complement each other, thereby improving the performance of the speaker verification model and reducing the equal error rate.
[0085] A speaker verification model training method provided by the present invention is described below in conjunction with specific embodiments.
[0086] Figure 1 FIG. 1 is a flow chart of the speaker verification model training method provided by the present invention. Figure 1 As shown, the speaker verification model training method provided in this embodiment includes:
[0087] S101, obtaining speech features corresponding to all speech samples in a training batch and label information corresponding to all speech samples.
[0088] Optionally, the speaker confirmation model training method provided by the present invention can be executed by an electronic device or a speaker confirmation model training device provided in the electronic device. The speaker confirmation model training device can be implemented by a combination of software and / or hardware. The speaker confirmation model training method provided by the present invention is described below with the execution subject being a speaker confirmation model training device.
[0089] All speech samples are a training batch.
[0090] Optionally, all voice samples may include multiple pieces of initial voice data from a voice open source dataset, may include multiple pieces of enhanced voice data after data enhancement processing is performed on the multiple pieces of initial voice data, or may include the above-mentioned multiple pieces of initial voice data and multiple pieces of enhanced voice data.
[0091] Optionally, the speech open source dataset may include at least one open source dataset of VoxCeleb1, VoxCeleb2, AISHELL-1, CN-Celeb, etc.
[0092] Optionally, the enhancement method used for data enhancement processing may be at least one of volume enhancement, reverberation enhancement, speed enhancement, pitch enhancement, noise enhancement, frequency domain masking, time domain masking, and the like.
[0093] In the case where all voice samples include multiple initial voice data and multiple enhanced voice data, if the multiple initial voice data include voice data of 7025 different speakers of VoxCeleb1 and VoxCeleb2, the multiple enhanced voice data can be obtained in the following way: based on a room acoustic impulse response (RIR) data set, reverberation processing is performed on the multiple initial voice data to obtain a first enhanced voice data set, and noise processing is performed on the multiple initial voice data to obtain a second enhanced voice data set.
[0094] Using a preset tool (such as an open source tool torchaudio), based on a first preset rate, the multiple pieces of initial speech data are subjected to an audio rate change process to obtain a third enhanced speech data set; based on a second preset rate, the multiple pieces of initial speech data are subjected to an audio rate change process to obtain a fourth enhanced speech data set;
[0095] Through a first preset algorithm (such as the SpecAugment algorithm), the voice data in the first enhanced voice data set, the second enhanced voice data set, the third enhanced voice data set and the fourth enhanced voice data set are randomly masked in the audio frame number dimension and the audio frequency dimension to obtain multiple enhanced voice data.
[0096] For example, the first preset rate may be 0.95, and the second preset rate may be 1.05.
[0097] For example, in the audio frame number dimension, the random value range of the mask width is [0, 10], and in the audio frequency dimension, the random value range of the mask width is [0, 8].
[0098] The label information corresponding to all voice samples is the speaker category corresponding to each voice sample.
[0099] The speech features corresponding to all speech samples are acoustic frame-level features.
[0100] S102, inputting the speech features into the global feature extraction network and the local feature extraction network in the initial speaker verification model respectively, to obtain target global features and target local features corresponding to the speech features.
[0101] Optionally, the local feature extraction network may be a time-delay neural network (TDNN).
[0102] Optionally, the global feature extraction network can be a Transformer network.
[0103] For a detailed description of the initial speaker verification model, see Figure 2 Example.
[0104] S103: Determine prediction information corresponding to all speech samples based on the target global features and the target local features.
[0105] Specifically, the speech features are respectively input into the global feature extraction network and the local feature extraction network in the initial speaker verification model to obtain the target global features and the target local features corresponding to the speech features, including:
[0106] Inputting the speech features into the global feature extraction network and the local feature extraction network in the initial speaker verification model respectively to obtain a first global feature and a first local feature;
[0107] Inputting the first global feature into a first data processing network in an initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature;
[0108] Inputting the second global feature into a local feature extraction network to obtain a second local feature and a target local feature;
[0109] Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions;
[0110] The third local feature is input into the global feature extraction network to obtain the target global feature, and the dimension of the third local feature is the same as that of the first global feature.
[0111] For details on how to obtain the second local feature, see Figure 3 For detailed description of obtaining the third local feature, please refer to Figure 4 Example.
[0112] S104: Update the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain a speaker verification model.
[0113] Optionally, a loss value between the label information and the prediction information is calculated, and based on the loss value, a model parameter of the initial speaker verification model is updated to obtain a speaker verification model.
[0114] In the present invention, speech features are input into the TDNN network and the Transformer network respectively, so that the TDNN network and the Transformer network learn the speech features in parallel, which solves the defect that the TDNN network and the Transformer network cannot learn in parallel in the related technology and improves the performance of the speaker confirmation model.
[0115] In some embodiments, obtaining speech features corresponding to all speech samples in a training batch includes:
[0116] Extract the Mel-Frequency Cepstral Coefficients (MFCCs) features of all speech samples;
[0117] The Mel-frequency cepstral coefficient features are input into the one-dimensional convolutional layer in the initial speaker verification model to obtain speech features.
[0118] Optionally, the process of obtaining the MFCCs features includes: performing pre-emphasis, framing, windowing, fast Fourier transform FFT, Mel filtering, and cepstrum analysis on the speech samples in sequence to obtain the MFCCs features.
[0119] Optionally, during the windowing process, a window length of 25 milliseconds and a window sliding length of 10 milliseconds are used.
[0120] The dimension of MFCCs features is 80.
[0121] The audio frequency dimension of the speech feature is 512, and the audio frame number dimension is T.
[0122] Specifically, the 80-dimensional MFCCs features are mapped into 512-dimensional MFCCs features through a one-dimensional convolutional layer to obtain a frame-level feature sequence (i.e., speech features).
[0123] In some embodiments, based on the target global features and the target local features, determining prediction information corresponding to all speech samples includes:
[0124] The global features of the target and the local features of the target are superimposed to obtain fused features;
[0125] The fused features are input into the Attentive Statistics Pooling (ASP) layer in the initial speaker verification model.
[0126] The output of the attention statistics pooling layer is input into the first batch of normalization layers in the initial speaker verification model to obtain the third normalized features;
[0127] Inputting the third normalized feature into a fully connected layer in the initial speaker verification model;
[0128] The output of the fully connected layer is input into the second batch normalization layer in the initial speaker verification model to obtain the fourth normalized feature;
[0129] The fourth normalized feature is input into the classifier in the initial speaker verification model to obtain prediction information.
[0130] For example, the classifier is an Additive angular margin softmax (AAM-softmax) classifier.
[0131] Based on the above embodiments, Figure 2 The initial speaker verification model is described.
[0132] Figure 2 Schematic diagram of the initial speaker confirmation model provided by the present invention. Figure 2 As shown, the initial speaker verification model includes: a one-dimensional convolutional layer (#0Conv1layer), a TDNN, a Transformer network, a first data processing network (IA), a second data processing network (IB), an adder, an ASP, a first batch of normalization layers, a fully connected layer, a second batch of normalization layers and a classifier.
[0133] TDNN includes: two one-dimensional convolutional layers (respectively: #1Conv1layer, #2Conv1layer), three residual network modules (respectively: #1SE-Res2block, #2SE-Res2block, #3SE-Res2block) and an adder.
[0134] The Transformer network includes: two one-dimensional convolutional layers (respectively: #1Conv1layer, #2Conv1layer) and three converter modules (respectively: #1TransBlock, #2TransBlock, #3TransBlock) and an adder.
[0135] The working process of the initial speaker verification model includes the following steps 1 to 11.
[0136] The first step is to obtain the MFCCs features and then input the MFCCs features into #0Conv1layer to obtain the speech features.
[0137] In the second step, the speech features are input into the #1Conv1layer in the TDNN, and the speech features are convolved through the #1Conv1layer to increase the nonlinearity of the model and obtain the first TDNN feature; the audio frequency dimension of the first TDNN feature is 512, and the audio frame number dimension is T. The speech features are input into the #1Conv1layer in the Transformer network, and the speech features are convolved (i.e., down-sampled) through the #1Conv1layer to obtain the first Transformer feature. The audio frequency dimension of the first Transformer feature is 256, and the audio frame number dimension is T / 3.
[0138] In the third step, the first TDNN feature is input into the #1SE-Res2block in the TDNN, and the local fine-grained features of the first TDNN feature are represented by the #1SE-Res2block to obtain the second TDNN feature. The first Transformer feature is input into the #1TransBlock in the Transformer network, and the global context features in the first Transformer feature are represented by the #1TransBlock to obtain the second Transformer feature.
[0139] In the fourth step, the second TDNN feature is input into the #2SE-Res2block in the TDNN, and its local fine-grained features are further represented by the #2SE-Res2block to obtain the first local feature. The second Transformer feature is input into the #2TransBlock in the Transformer network to further represent the global context feature to obtain the first global feature.
[0140] Step 5: Input the first global feature into IA (see Figure 3 Embodiment) is processed to obtain the second global feature.
[0141] In the sixth step, the second global feature and the first local feature are input into the adder for superposition processing, and then input into the #3se-Res2Block in TDNN to obtain the second local feature.
[0142] Step 7: Input the second local feature into IB (see Figure 4 Embodiment) is processed to obtain the third local feature.
[0143] In the eighth step, the third local feature and the first global feature are input into the adder for superposition processing; the feature obtained after the superposition processing is input into the #3TransBlock in the Transformer network to obtain the third Transformer feature.
[0144] In the ninth step, the second TDNN feature, the first local feature and the second local feature are concatenated in the frame dimension and input into the #2Conv1layer in TDNN to obtain the target local feature.
[0145] In the tenth step, the second Transformer feature, the first global feature, and the third Transformer feature are concatenated in the frame dimension, and the concatenated features are input into the #2Conv1layer in the Transformer network to obtain the target global feature.
[0146] In the eleventh step, the target global features and the target local features are input into the adder for superposition processing to obtain the fused features.
[0147] In some embodiments, inputting the first global feature into a first data processing network in an initial speaker verification model to obtain a second global feature includes:
[0148] Performing upsampling processing on the first global feature through a first data processing network to obtain an upsampled feature, wherein the upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension;
[0149] Performing convolution processing on the upsampled features through a first data processing network to obtain first convolution features, wherein the first convolution features and the upsampled features have the same audio frame number dimension but different audio frequency dimensions;
[0150] Performing batch normalization processing on the first convolutional features through a first data processing network to obtain first normalized features;
[0151] The product of the first normalized feature and the first preset adjustable parameter is calculated through the first data processing network to obtain the second global feature.
[0152] Figure 3 Schematic diagram of the structure of the first data processing network provided by the present invention. Figure 3 As shown, the first data processing network includes: an upsampling module, a convolution module (Conv1layer), a batch normalization (BN) module and a multiplier.
[0153] The upsampling module is used to upsample the first global feature to expand its audio frame number dimension to obtain an upsampled feature. The upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension.
[0154] For example, the audio frame number dimension of the first global feature is T / 3, and the audio frequency dimension is 256; the audio frame number dimension of the upsampling feature is T, and the audio frequency dimension is 256.
[0155] Conv1layer is used to expand the audio frequency dimension of speech features. Optionally, the relevant parameters of the convolution module include the convolution kernel size k, the step size s and the number of channels c.
[0156] Specifically, Conv1layer is used to perform convolution processing on the upsampled feature based on parameters k, s, and c to expand the audio frequency dimension of the upsampled feature, thereby obtaining the first convolution feature. The upsampled feature and the first convolution feature have the same audio frame number dimension and different audio frequency dimensions. For example, when k=3, s=1, and c=512, if the audio frequency dimension of the upsampled feature is 256 and the audio frame number dimension is T, then the audio frequency dimension of the first convolution feature is 512 and the audio frame number dimension is T.
[0157] The BN module is used to perform batch normalization on the first convolutional feature to obtain the first normalized feature.
[0158] The multiplier is used to multiply the first normalized feature by a first preset adjustable parameter (PA) to obtain a second global feature.
[0159] In the present invention, a second global feature having the same audio frequency dimension and audio frame number dimension as the first local feature is obtained through a first data processing module, so that the second global feature and the first local feature can be superimposed on each other through an adder, thereby achieving the purpose of TDNN being able to learn the global features of the Transformer network during the learning process, and improving the accuracy of the speaker confirmation model results.
[0160] In some embodiments, the second local feature is input into a second data processing network in the initial speaker verification model to obtain a third local feature, including:
[0161] The second local feature is convolved through a second data processing network to obtain a second convolution feature; the second local feature and the second convolution feature have different audio frequency dimensions and the same audio frame number dimension;
[0162] Performing layer normalization processing on the second convolutional features through a second data processing network to obtain second normalized features;
[0163] The product of the second normalized feature and the second preset adjustable parameter is calculated through the second data processing network to obtain a third global feature.
[0164] For a detailed description of the second data processing network, see Figure 4 .
[0165] Figure 4 Schematic diagram of the structure of the second data processing network provided by the present invention. Figure 4As shown, the second data processing network includes: a convolution module (Conv1layer)), a layer normalization (LN) module and a multiplier.
[0166] Conv1layer is used to perform convolution processing on the second local feature based on parameters k, s, and c to reduce the audio frequency dimension and audio frame number dimension of the second local feature, thereby obtaining the second convolution feature. Among them, the audio frame number dimension and audio frequency dimension of the second local feature and the second convolution feature are different. For example, when k=3, s=2, and c=256, if the audio frequency dimension of the second local feature is 512 and the audio frame number dimension is T, then the audio frequency dimension of the second convolution feature is 256 and the audio frame number dimension is T / 3.
[0167] The LN module is used to perform layer normalization on the second convolutional feature to obtain the second normalized feature.
[0168] The multiplier is used to multiply the second normalized feature by a second preset adjustable parameter (PB) to obtain a third local feature.
[0169] It should be noted that, during the model training process, the second preset adjustable parameter is a preset adjustable parameter that can be updated.
[0170] In the present invention, a third local feature having the same audio frequency dimension and audio frame number dimension as the first global feature is obtained through a second data processing module, so that the third local feature and the first global feature can be superimposed on each other through an adder, thereby enabling the Transformer network to learn the local features of TDNN during the learning process, thereby improving the accuracy of the speaker confirmation model results.
[0171] In some embodiments, updating the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain the speaker verification model includes:
[0172] Calculate the label smoothing crossentropy loss between label information and prediction information;
[0173] Based on the smoothed cross entropy loss value, the model parameters of the initial speaker verification model are updated to obtain the speaker verification model.
[0174] The label information may be reference category information obtained based on all speech samples.
[0175] The smoothed cross entropy loss value may be a loss value obtained by using a cross entropy loss function to measure the degree of fit of the predicted information to the label information.
[0176] Optionally, in the process of updating the model parameters of the initial speaker verification model, an Adam optimizer may be used for back propagation optimization.
[0177] The speaker confirmation model training device provided by the present invention is described below. The speaker confirmation model training device described below and the speaker confirmation model training method described above can be referred to each other.
[0178] Figure 5 Schematic diagram of the structure of the speaker verification model training device provided by the present invention. Figure 5 As shown, the speaker verification model training device includes:
[0179] An acquisition module 510 is used to acquire speech features corresponding to all speech samples in a training batch and label information corresponding to all speech samples;
[0180] A feature extraction module 520 is used to input the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model, respectively, to obtain a target global feature and a target local feature corresponding to the speech features;
[0181] A determination module 530, configured to determine prediction information corresponding to all speech samples based on the target global feature and the target local feature;
[0182] The updating module 540 is used to update the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain the speaker verification model. According to the speaker verification model training device provided by the present invention, the feature extraction module 520 is specifically used to:
[0183] Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain a first global feature and a first local feature;
[0184] Inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature;
[0185] Inputting the second global feature into the local feature extraction network to obtain a second local feature and the target local feature;
[0186] Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions;
[0187] The third local feature is input into the global feature extraction network to obtain a target global feature, and the dimension of the third local feature is the same as that of the first global feature.
[0188] According to a speaker verification model training device provided by the present invention, the feature extraction module 520 is specifically used for:
[0189] Performing upsampling processing on the first global feature through the first data processing network to obtain an upsampled feature, wherein the upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension;
[0190] Performing convolution processing on the upsampled feature through the first data processing network to obtain a first convolution feature, wherein the first convolution feature and the upsampled feature have the same audio frame number dimension and a different audio frequency dimension;
[0191] Performing batch normalization processing on the first convolutional features through the first data processing network to obtain first normalized features;
[0192] The product of the first normalized feature and a first preset adjustable parameter is calculated through the first data processing network to obtain the second global feature.
[0193] According to a speaker verification model training device provided by the present invention, the feature extraction module 520 is specifically used for:
[0194] The second local feature is convolved by the second data processing network to obtain a second convolution feature; the second local feature and the second convolution feature have different audio frequency dimensions and different audio frame number dimensions;
[0195] Performing layer normalization processing on the second convolutional features through the second data processing network to obtain second normalized features;
[0196] The product of the second normalized feature and a second preset adjustable parameter is calculated through the second data processing network to obtain the third local feature.
[0197] According to a speaker verification model training device provided by the present invention, the acquisition module 510 is specifically used for:
[0198] Extracting Mel-frequency cepstral coefficient features of all speech samples;
[0199] The Mel frequency cepstral coefficient feature is input into a one-dimensional convolutional layer in the initial speaker verification model to obtain the speech feature.
[0200] According to a speaker verification model training device provided by the present invention, the determination module 530 is specifically used for:
[0201] Superimposing the target global feature and the target local feature to obtain a fused feature;
[0202] Inputting the fused features into the attention statistics pooling layer in the initial speaker verification model;
[0203] Inputting the output result of the attention statistics pooling layer into the first batch of normalization layers in the initial speaker verification model to obtain a third normalized feature;
[0204] Inputting the third normalized feature into a fully connected layer in the initial speaker verification model;
[0205] Inputting the output result of the fully connected layer into the second batch normalization layer in the initial speaker verification model to obtain fourth normalized features;
[0206] The fourth normalized feature is input into a classifier in the initial speaker verification model to obtain the prediction information.
[0207] According to a speaker verification model training device provided by the present invention, the updating module 540 is specifically used for:
[0208] Calculating a smoothed cross entropy loss value between the label information and the prediction information;
[0209] Based on the smoothed cross entropy loss value, model parameters of the initial speaker verification model are updated to obtain a speaker verification model.
[0210] Figure 6 Schematic diagram of the physical structure of the electronic device provided by the present invention. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the speaker confirmation model training method, which includes: obtaining speech features corresponding to all speech samples in a training batch and label information corresponding to all speech samples; inputting the speech features into the global feature extraction network and the local feature extraction network in the initial speaker confirmation model respectively to obtain the target global features and the target local features corresponding to the speech features; determining the prediction information corresponding to all speech samples based on the target global features and the target local features; updating the model parameters of the initial speaker confirmation model according to the label information and the prediction information to obtain the speaker confirmation model.
[0211] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0212] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speaker confirmation model training method provided by the above methods, which includes: obtaining speech features corresponding to all speech samples in a training batch, and label information corresponding to all speech samples; inputting the speech features into the global feature extraction network and the local feature extraction network in the initial speaker confirmation model respectively to obtain target global features and target local features corresponding to the speech features; based on the target global features and the target local features, determining the prediction information corresponding to all speech samples; and updating the model parameters of the initial speaker confirmation model according to the label information and the prediction information to obtain a speaker confirmation model.
[0213] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speaker confirmation model training method provided by the above-mentioned methods, the method comprising: obtaining speech features corresponding to all speech samples in a training batch, and label information corresponding to all the speech samples; inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker confirmation model respectively, to obtain target global features and target local features corresponding to the speech features; based on the target global features and the target local features, determining the prediction information corresponding to all the speech samples; and updating the model parameters of the initial speaker confirmation model according to the label information and the prediction information to obtain a speaker confirmation model.
[0214] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0215] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A speaker verification model training method, characterized in that: include: Obtaining speech features corresponding to all speech samples in a training batch and label information corresponding to all speech samples; The speech features are respectively input into a global feature extraction network and a local feature extraction network in an initial speaker verification model to obtain target global features and target local features corresponding to the speech features, wherein the global feature extraction network is a Transformer network, and the local feature extraction network is a TDNN network; Determining prediction information corresponding to all speech samples based on the target global feature and the target local feature; updating the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain a speaker verification model; The step of inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model to obtain target global features and target local features corresponding to the speech features comprises: Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain a first global feature and a first local feature; Inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature; Inputting the second global feature into the local feature extraction network to obtain a second local feature, and obtaining the target local feature according to the first local feature and the second local feature; Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions; The third local feature is input into the global feature extraction network, and the target global feature is obtained according to the third local feature and the first global feature, wherein the third local feature has the same dimension as the first global feature.
2. The speaker verification model training method according to claim 1, characterized in that: The step of inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature comprises: Performing upsampling processing on the first global feature through the first data processing network to obtain an upsampled feature, wherein the upsampled feature and the first global feature have different audio frame number dimensions and the same audio frequency dimension; Performing convolution processing on the upsampled feature through the first data processing network to obtain a first convolution feature, wherein the first convolution feature and the upsampled feature have the same audio frame number dimension and a different audio frequency dimension; Performing batch normalization processing on the first convolutional features through the first data processing network to obtain first normalized features; The product of the first normalized feature and a first preset adjustable parameter is calculated through the first data processing network to obtain the second global feature.
3. The speaker verification model training method according to claim 2, characterized in that: The step of inputting the second local feature into the second data processing network in the initial speaker verification model to obtain the third local feature comprises: The second local feature is convolved by the second data processing network to obtain a second convolution feature; the second local feature and the second convolution feature have different audio frequency dimensions and different audio frame number dimensions; Performing layer normalization processing on the second convolutional features through the second data processing network to obtain second normalized features; The product of the second normalized feature and a second preset adjustable parameter is calculated through the second data processing network to obtain the third local feature.
4. The speaker verification model training method according to any one of claims 1 to 3, characterized in that: The obtaining of speech features corresponding to all speech samples in a training batch includes: Extracting Mel-frequency cepstral coefficient features of all speech samples; The Mel frequency cepstral coefficient feature is input into a one-dimensional convolutional layer in the initial speaker verification model to obtain the speech feature.
5. The speaker verification model training method according to any one of claims 1 to 3, characterized in that: The determining, based on the target global feature and the target local feature, prediction information corresponding to all speech samples includes: Superimposing the target global feature and the target local feature to obtain a fused feature; Inputting the fused features into the attention statistics pooling layer in the initial speaker verification model; Inputting the output result of the attention statistics pooling layer into the first batch of normalization layers in the initial speaker verification model to obtain a third normalized feature; Inputting the third normalized feature into a fully connected layer in the initial speaker verification model; Inputting the output result of the fully connected layer into the second batch normalization layer in the initial speaker verification model to obtain fourth normalized features; The fourth normalized feature is input into a classifier in the initial speaker verification model to obtain the prediction information.
6. The speaker verification model training method according to any one of claims 1 to 3, characterized in that: The updating of the model parameters of the initial speaker verification model according to the label information and the prediction information to obtain the speaker verification model includes: Calculating a smoothed cross entropy loss value between the label information and the prediction information; Based on the smoothed cross entropy loss value, model parameters of the initial speaker verification model are updated to obtain a speaker verification model.
7. A speaker verification model training device, characterized in that: include: An acquisition module, used to acquire speech features corresponding to all speech samples in a training batch, and label information corresponding to all speech samples; A feature extraction module, used to input the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model, respectively, to obtain target global features and target local features corresponding to the speech features, wherein the global feature extraction network is a Transformer network, and the local feature extraction network is a TDNN network; A determination module, configured to determine prediction information corresponding to all speech samples based on the target global feature and the target local feature; An updating module, configured to update the model parameters of the initial speaker confirmation model according to the label information and the prediction information to obtain a speaker confirmation model; The feature extraction module is specifically used for: Inputting the speech features into a global feature extraction network and a local feature extraction network in an initial speaker verification model respectively to obtain a first global feature and a first local feature; Inputting the first global feature into a first data processing network in the initial speaker verification model to obtain a second global feature, wherein the first global feature and the second global feature have different dimensions, and the second global feature has the same dimension as the first local feature; Inputting the second global feature into the local feature extraction network to obtain a second local feature, and obtaining the target local feature according to the first local feature and the second local feature; Inputting the second local feature into a second data processing network in the initial speaker verification model to obtain a third local feature, wherein the second local feature and the third local feature have different dimensions; The third local feature is input into the global feature extraction network, and the target global feature is obtained according to the third local feature and the first global feature, wherein the third local feature has the same dimension as the first global feature.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speaker verification model training method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speaker verification model training method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the speaker verification model training method according to any one of claims 1 to 6 is implemented.