Training methods, systems, media, and cross-channel and dialect voiceprint recognition models
By introducing a text feature extraction network and a multi-task loss function into the X-vector model, the problem of model shift under cross-channel and cross-dialect conditions is solved, achieving higher recognition accuracy and robustness.
Patent Information
- Application Number
- CN202211221436.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-08
- Publication Date
- 2026-01-20
- Estimated Expiration
- 2042-10-08
AI Technical Summary
Speaker recognition models based on the X-vector architecture are prone to shifts due to channel and dialect mismatches when the amount of training data increases, which affects the recognition performance.
By constructing a voiceprint recognition model that sequentially connects a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer, and a first output layer, a text feature extraction network is introduced and a multi-task loss function is adopted. The training process integrates speech and text information to reduce the impact of channel and dialect mismatch.
It improves the model's generalization and robustness under cross-channel and cross-dialect conditions, reduces the accumulation of adverse factors caused by the increase in training data, and improves recognition accuracy.
Smart Images
Figure CN115762479B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of speaker recognition, in particular to a training method, system, medium and voiceprint recognition model across channels and dialects. BACKGROUND
[0002] The main advantage of the current speaker recognition model based on X-vector framework is that, in the case of sufficient training data, it has higher recognition rate than the traditional i-vector system. However, when it comes to cross-channel speech recognition, the defects caused by the model architecture are also more obvious. The main problem is that as the amount of training data increases, the trained model will deviate from the channel difference in the training data, resulting in a final training result that is completely different from the expected result.
[0003] The above-mentioned defects are mainly due to the fact that the current model based on X-vector framework mainly classifies multiple speakers by using the cross-entropy of some feature vectors in the speaker's voice data, and uses pre-labeled speaker labels to reduce the entropy value of the loss function to realize the training of the model. Since the speaker recognition model based on X-vector framework requires a large amount of training data to achieve higher recognition rate, it is inevitable that when the amount of training data increases, the adverse factors (such as channel mismatch, dialect mismatch, etc.) in the training samples will be highlighted. When the model is trained, these adverse factors will cause the overall model to deviate from the adverse factors, and finally will have a negative impact on the overall effect.
[0004] Among them, the above-mentioned cross-channel refers to the channel source of the voice data, that is, the difference in the channel caused by different sound collection devices. SUMMARY
[0005] The present application provides a training method for a voiceprint recognition model across channels and dialects, which can overcome some or some of the defects of the prior art.
[0006] According to the training method for the voiceprint recognition model across channels and dialects of the present application, the voiceprint recognition model has a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer and a first output layer connected in sequence, the first input layer is used to obtain a voice feature sequence W i 1 The voiceprint feature extraction network layer is used to output a voice feature sequence W i * The first output layer is used to output a speaker prediction sequence
[0007] When training the voiceprint recognition model, the following steps are included:
[0008] Step S1, constructing a text feature extraction network, a second output layer and a third output layer;
[0009] The text feature extraction network is configured to obtain a text feature sequence T i 1 and output the text representation sequence T i * The second output layer is connected to the feedforward network layer and configured to output a channel prediction sequence The third output layer is connected to the feedforward network layer and configured to output a text prediction sequence
[0010] Step S2, constructing a training set X;
[0011] X = {X i |i = 1, 2, 3, … N}, X i = {S i , C i , W i 0 , T i 0}; N is the total number of samples, X i is the i-th sample, S i , C i , W i 0 and T i 0 are respectively the speaker label, channel label, speech data and text data of the sample X i ;
[0012] Step S3, initializing parameters of the voiceprint recognition model;
[0013] Step S4, training the voiceprint recognition model and constructing a loss function L;
[0014] In the training of the voiceprint recognition model, the speech representation sequence W i * and the text representation sequence T i * are fused to obtain a fusion feature sequence A i , and then the fusion feature sequence A i is taken as the output of the feedforward network layer to obtain a speaker prediction sequence a channel prediction sequence and a text prediction sequence
[0015] Any feature value in the fusion feature sequence A i is the speech representation sequence W i * and the text representation sequence T i* an arithmetic sum of values of corresponding features;
[0016] wherein, L = -a2L2 + a1L1 + a3L3; L1, L2 and L3 are loss functions for representing the difference between the predicted value and the real value of the channel, the speaker and the text data respectively, a1, a2 and a3 are weights of the loss functions L1, L2 and L3 respectively, the value range of a1, a2 and a3 is (0, 1) and is artificially set;
[0017] Step S5, updating the parameters of the voiceprint recognition model;
[0018] Step S6, repeating steps S4 and S5 until the loss function is optimal, completing the training of the voiceprint recognition model.
[0019] Through the above, when the voiceprint recognition model is trained, the text information can be introduced by the text feature extraction network, and the channel source, the speaker source and the text information can be predicted based on the first input layer, the second input layer and the third input layer, and then the loss function of the whole model training is constructed based on the loss functions of the three; Therefore, the channel recognition and the text recognition can be introduced in the training process, and the strong robustness of the trained model can be achieved in a multi-task manner, thereby reducing the accumulation of adverse factors such as channel mismatch and dialect mismatch caused by the increase of training data.
[0020] As a preferred, the text feature extraction network has a second input layer and a second convolutional layer connected in sequence, the second input layer is used for performing vectorization processing on the text data T i 0 to obtain a vectorized text feature sequence T i 1 , and the second convolutional layer is used for performing convolution processing on the text feature sequence T i to obtain a text representation sequence T i * . Therefore, the processing of the text data can be preferably realized to ensure the realization of the subsequent steps.
[0021] As a preferred, the second input layer has 2 layers of bidirectional LSTM network layers, and is used for processing the text data T i 0 into a text vector with a dimension of 512. Therefore, the acquisition of the text feature sequence T i 1 can be preferably realized.
[0022] As preferred, the second convolutional layer and the first convolutional layer have the same architecture and share parameters. By ensuring that they have the same architecture, the implementation of data fusion in step S4 is ensured; by ensuring the sharing of parameters, the performance of the final trained voiceprint recognition model is ensured from the interference caused by the differences in parameters between the second convolutional layer and the first convolutional layer in the iteration process.
[0023] As preferred, the first output layer is used to extract the MFCC features of the speech data W i 0 and output the speech feature sequence W i 1 . Thus, the acquisition of the speech feature sequence W i 1 is preferably achieved.
[0024] As preferred, the voiceprint feature extraction network layer adopts a transformer network. Thus, the recognition and processing of speech data can be preferably achieved by means of the existing architecture.
[0025] As preferred, the loss functions L1, L2 and L3 all adopt a cross-entropy loss function. Thus, it is convenient to implement.
[0026] In addition, the present application also provides a voiceprint recognition model training device across channels and dialects, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of any one of the above-mentioned training methods. Thus, the above-mentioned training method can be preferably achieved.
[0027] In addition, the present application also provides a computer readable medium having a computer program stored thereon, characterized in that: the computer program is executed to realize the steps of any one of the above-mentioned training methods. Thus, the above-mentioned training method can be preferably achieved.
[0028] In addition, the present application also provides a voiceprint recognition model across channels and dialects, which has a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer and a first output layer connected in sequence, the first input layer is used to acquire the speech feature sequence W i 1 , the voiceprint feature extraction network layer is used to output the speech representation sequence W i * , and the first output layer is used to output the speaker prediction sequence which is obtained by training by any one of the above-mentioned training methods. Thus, when facing the processing of cross-channel and multi-dialect speech data, it can have better generalization and robustness. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 is a schematic diagram of the framework of the training method in embodiment 1. Detailed Implementation
[0030] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.
[0031] Example 1
[0032] Seen in Figure 1 This embodiment provides a training method for a cross-channel and dialect voiceprint recognition model. The voiceprint recognition model has a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer, and a first output layer connected in sequence. The first input layer is used to acquire the speech feature sequence W. i 1 The voiceprint feature extraction network layer is used to output the speech representation sequence W. i * The first output layer is used to output the speaker prediction sequence.
[0033] The training of this voiceprint recognition model includes the following steps:
[0034] Step S1: Construct a text feature extraction network, a second output layer, and a third output layer;
[0035] Among them, the text feature extraction network is used to obtain the text feature sequence T. i 1 And output the text representation sequence T i * The second output layer is connected to the feedforward network layer and is used to output the channel prediction sequence. The third output layer is connected to the feedforward network layer and is used to output the text prediction sequence.
[0036] Step S2: Construct the training set X;
[0037] Where, X = {X} i |i=1,2,3,…N},X i ={S i C i W i 0 T i 0}; N is the total number of samples, X i For the i-th sample, S i C i W i 0 and T i 0 Sample X ispeaker labels, channel labels, speech data and text data;
[0038] Step S3, initializing parameters of the voiceprint recognition model;
[0039] Step S4, training the voiceprint recognition model and constructing a loss function L;
[0040] When training the voiceprint recognition model, the speech representation sequence W i * and the text representation sequence T i * are fused and a fused feature sequence A i is obtained, and then the fused feature sequence A i is taken as the output of the feedforward network layer to obtain a speaker prediction sequence a channel prediction sequence and a text prediction sequence
[0041] wherein the value of any feature in the fused feature sequence A i is the arithmetic sum of the values of the corresponding features in the speech representation sequence W i * and the text representation sequence T i *
[0042] wherein L = -a2L2 + a1L1 + a3L3; L1, L2 and L3 are respectively loss functions for representing the difference between the predicted values and the true values of the channel, speaker and text data, a1, a2 and a3 are respectively the weights of the loss functions L1, L2 and L3, the value range of a1, a2 and a3 is (0, 1) and is artificially set;
[0043] Step S5, updating the parameters of the voiceprint recognition model;
[0044] Step S6, repeating steps S4 and S5 until the loss function is optimal, and completing the training of the voiceprint recognition model.
[0045] Through the above, when training the voiceprint recognition model, the text information can be additionally introduced through the text feature extraction network, and the channel source, the speaker source and the text information can be simultaneously predicted based on the first input layer, the second input layer and the third input layer, and then the loss function of the three is used to construct the loss function of the entire model training. Therefore, the channel recognition and the text recognition can be preferably introduced in the training process, and the strong robustness of the trained model can be achieved in a multi-task manner, thereby minimizing the accumulation of adverse factors such as channel mismatch and dialect mismatch caused by the increase of the training data volume.
[0046] The voiceprint recognition model obtained based on the training method can preferably realize the generalization and robustness of speaker recognition in the cross-channel and cross-dialect situation.
[0047] It can be understood that the text feature extraction network, the second output layer and the third output layer are only in the training process, and the input of the trained voiceprint recognition model can only be the voice data to be recognized and the output can only be the predicted speaker.
[0048] In addition, it can be understood that the training set X in the embodiment should ensure enough samples of different channels and different dialects when constructing.
[0049] In the embodiment, the text feature extraction network has a second input layer and a second convolutional layer connected in sequence, the second input layer is used to perform vectorization processing on the text data T i 0 to obtain a vectorized text feature sequence T i 1 , and the second convolutional layer is used to perform convolution processing on the text feature sequence T i to obtain a text representation sequence T i * . Therefore, the processing of the text data can be preferably realized to ensure the realization of the subsequent steps.
[0050] In the embodiment, the second input layer has two bidirectional LSTM network layers and is used to process the text data T i 0 into a text vector with a dimension of 512. Therefore, the text feature sequence T i 1 can be preferably obtained.
[0051] In the embodiment, the second convolutional layer and the first convolutional layer have the same architecture and share parameters. By ensuring the same architecture, the realization of data fusion in step S4 is ensured; by ensuring the sharing of parameters, the performance of the voiceprint recognition model obtained by the final training is ensured. The interference caused by the difference in parameters between the second convolutional layer and the first convolutional layer in the iteration process.
[0052] In the embodiment, the first output layer is used to extract the MFCC features of the voice data W i 0 and serve as a voice feature sequence W i 1 . Therefore, the voice feature sequence W i 1 can be preferably obtained.
[0053] In this embodiment, the voiceprint feature extraction network layer adopts a transformer network. Therefore, the voice data can be better recognized and processed by means of the existing architecture.
[0054] In this embodiment, the loss functions L1, L2 and L3 all adopt cross-entropy loss functions. Therefore, it is convenient to implement.
[0055] Based on the training method of this embodiment, the present embodiment further provides a training device of a cross-channel and dialect voiceprint recognition model, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of the above training method. Therefore, the above training method can be better implemented.
[0056] Based on the training method of this embodiment, the present embodiment further provides a computer readable medium having a computer program stored thereon, which is executed to realize the steps of the above training method. Therefore, the above training method can be better implemented.
[0057] In addition, based on the training method of this embodiment, the present embodiment further provides a cross-channel and dialect voiceprint recognition model, which has a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer and a first output layer connected in sequence, the first input layer is used to obtain a speech feature sequence W i 1 , the voiceprint feature extraction network layer is used to output a speech representation sequence W i * , and the first output layer is used to output a speaker prediction sequence which is obtained by training the above training method. Therefore, when facing the processing of cross-channel and multi-dialect speech data, it can have better generalization and robustness.
[0058] The above describes the present application and its embodiments in a schematic manner, which is not restrictive, and the embodiments shown in the drawings are only one of the embodiments of the present application, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired thereby, without departing from the spirit of the present application, similar structural modes and embodiments can be designed without creative design, which shall belong to the protection scope of the present application.
Claims
1. A method for training a cross-channel and dialect voiceprint recognition model, the voiceprint recognition model having a first input layer, a first convolutional layer, a voiceprint feature extraction network layer, a feedforward network layer, and a first output layer connected in sequence, the first input layer being configured to obtain a speech feature sequence , the voiceprint feature extraction network layer being configured to output a speech representation sequence , and the first output layer being configured to output a speaker prediction sequence . When training the voiceprint recognition model, the following steps are included: Step S1, constructing a text feature extraction network, a second output layer and a third output layer; wherein The text feature extraction network is configured to obtain a sequence of text features and output a sequence of text representations , a second output layer connected to the feedforward network layers and configured to output a sequence of channel predictions , and a third output layer connected to the feedforward network layers and configured to output a sequence of text predictions ; Step S2, constructing a training set X; in, , N is the total number of samples. For the i-th sample, , , and Samples Speaker tags, channel tags, voice data, and text data; Step S3, initializing parameters of the voiceprint recognition model; Step S4, training the voiceprint recognition model and constructing a loss function L; In training the voiceprint recognition model, the speech representation sequence and the text representation sequence are fused to obtain a fused feature sequence , and then the fused feature sequence is taken as the output of a feedforward network layer to further obtain a speaker prediction sequence , a channel prediction sequence and a text prediction sequence ; wherein the value of any feature in the sequence of fusion features is the sequence of speech features and the sequence of text features is the arithmetic sum of the values of the corresponding features in the sequence of fusion features wherein, ; , and are loss functions for characterizing the difference between the predicted values and the true values of the channel, speaker and text data, respectively, , and are weights of the loss functions , and , , and all have a value range of (0, 1) and are artificially set; Step S5, updating the parameters of the voiceprint recognition model; Step S6, repeating steps S4 and S5 until the loss function is optimal, and completing the training of the voiceprint recognition model. 2.The method of claim 1, wherein: The text feature extraction network has a second input layer and a second convolutional layer connected in turn, the second input layer is used for performing vectorization processing on the text data to obtain a vectorized text feature sequence , and the second convolutional layer is used for performing convolution processing on the text feature sequence to obtain a text representation sequence . 3.The method of claim 2, wherein: The second input layer has a 2-layer bidirectional LSTM network layer and is used to process the text data into a text vector with 512 dimensions. 4.The method of claim 2, wherein: The second convolutional layer and the first convolutional layer have the same architecture and share parameters. 5.The method of claim 1, wherein: The first output layer is configured to extract MFCC features of the speech data and output the MFCC features as a sequence of speech features. . 6.The method of claim 1, wherein: The voiceprint feature extraction network layer adopts a transformer network. 7.The method of claim 1, wherein: Loss function , and All employ a cross-entropy loss function. 8.A device for training a cross-channel and dialect voiceprint recognition model, comprising a memory and a processor, wherein the memory stores a computer program, and the device is characterized in that: The processor, when executing the computer program, implements the steps of the training method of any one of claims 1 to 7.
9. A computer readable medium having stored thereon a computer program, characterized in that: The computer program, when executed, implements the steps of the training method of any one of claims 1 to 7.