An Automatic Mongolian Speech Quality Assessment Method Based on Hierarchical Transfer Learning

Abstract

The present invention discloses a Mongolian automatic speech quality assessment method based on hierarchical transfer learning, which includes the following steps: pre-training an English speech self-supervised model and an English speech quality assessment model to obtain a trained English speech self-supervised model and a trained English speech quality assessment model; performing transfer learning on the trained English speech self-supervised model and the trained English speech quality assessment model to obtain a trained self-supervised model and a trained speech quality assessment model; using the trained speech self-supervised model and the BERT model to extract the feature vectors of Mongolian speech and the text features in the corresponding text; fusing the speech features and the text features into sentence-level semantic features f, and sending f into the trained speech quality assessment model to obtain the MOS score z corresponding to the speech signal, completing the automatic assessment of Mongolian speech quality, pioneering an automatic assessment method for Mongolian speech quality and filling the gap in this field.

Description

Technical Field

[0001] The invention relates to the field of voice communication, and in particular to a Mongolian automatic voice quality assessment method based on hierarchical transfer learning. Background Art

[0002] In the field of speech generation, speech quality evaluation is an important indicator for evaluating the quality of speech generated by a model. Among them, speech quality evaluation related to naturalness is an indicator that the industry pays more attention to. However, the manual evaluation method requires inviting a large number of volunteers to evaluate the generated speech samples, which is a time-consuming and labor-intensive process. Therefore, the automatic speech quality evaluation method based on deep learning models has received great attention from the academic community. Traditional speech quality evaluation methods mainly use the following three models:

[0003] (1) Speech quality assessment model based on convolutional neural network: This method uses a multi-layer convolutional neural network to build a speech quality assessment model, taking the mel-spectrogram feature of speech as input and the subjective mean opinion score (MOS) corresponding to the speech signal as output to update the network parameters. However, speech signals are time series signals, and convolutional neural networks cannot model the time series characteristics of speech signals.

[0004] (2) Speech quality assessment model based on recurrent neural network: This method uses a bidirectional recurrent neural network to construct a speech quality assessment model. It also uses the mel-spectrogram features of speech as input and the subjective MOS score corresponding to the speech signal as output to learn the network parameters.

[0005] (3) Speech quality assessment model based on convolutional-recurrent neural network: In order to further combine the advantages of convolutional neural network and recurrent neural network in speech feature modeling, a speech quality assessment model based on convolutional-recurrent neural network is proposed. This model learns the global semantic knowledge in the spectral features of speech signals and the temporal features of speech signals to accurately predict MOS scores.

[0006] The above methods use the powerful modeling ability of neural networks to automatically predict the quality of speech and evaluate the speech quality by taking speech feature parameters as input. However, the current methods are mainly aimed at mainstream languages such as English. The difficulty of obtaining speech MOS scores for mainstream languages is relatively low, and the data scale is large. The model parameters trained under large-scale data conditions can show good evaluation performance. However, when evaluating Mongolian speech quality, due to the limitation of data scale, neural networks with the same structure often find it difficult to achieve satisfactory performance. Summary of the invention

[0007] In view of the above deficiencies in the prior art, the Mongolian automatic speech quality evaluation method based on hierarchical transfer learning provided by the present invention solves the problem of poor effect of automatically evaluating the quality of Mongolian speech in the prior art.

[0008] In order to achieve the above object of the invention, the technical solution adopted by the present invention is: to provide a Mongolian automatic speech quality evaluation method based on hierarchical transfer learning, including the following steps:

[0009] S1. Pretrain an English speech self-supervised model and an English speech quality evaluation model to obtain a trained English speech self-supervised model and a trained English speech quality evaluation model;

[0010] S2. Perform transfer learning on the trained English speech self-supervised model and the trained English speech quality evaluation model to obtain a trained Mongolian speech self-supervised model and a trained Mongolian speech quality evaluation model;

[0011] S3. Use the trained Mongolian speech self-supervised model and the BERT model to extract the feature vector fs of the Mongolian speech and the text feature ft in the corresponding text respectively;

[0012] S4. Fuse fs and ft into a sentence-level semantic feature f, and send f into the trained Mongolian speech quality evaluation model to obtain the MOS score z corresponding to the Mongolian speech signal, and complete the automatic evaluation of the quality of Mongolian speech.

[0013] Further: The step S1 includes the following sub-steps:

[0014] S11. Read the English speech signal and mask the Mel spectrogram feature f of 15% of the speech frames mel , as the input sequence xe of the English speech self-supervised model;

[0015] S12. Read the input sequence xe through a deep bidirectional Transformer encoder, calculate the loss function L MLM , and perform backpropagation on it to complete the MLM task and obtain a trained English speech self-supervised model, where the loss function L MLM The mathematical expression of is:

[0016]

[0017] Among them, θ is the parameter of the bidirectional Transformer, θ1 is the parameter of the output layer connected by the bidirectional Transformer in the MLM task,, M represents the set of masked words, p(.) represents the conditional probability, m represents the current frame, m i represents the i-th speech frame, and V is the number of speech frames;

[0018] S13. Input the Mel-spectrum features f of the English speech signal mel into the CNN layer, and process f through the CNN layer mel to obtain the local and global information h1 of f, where the mathematical expression of h1 is: mel h1 = conv1(W, f

[0019] ) + b mel where conv1(,) is the convolution operation, W is the weight matrix, and b is the bias value;

[0020]

[0021] S14. Input the local and global information h1 into the bidirectional long short-term memory network to obtain the true value z' of MOS, and calculate the predicted value z” of the MOS score, where the expression of z” is:

[0022] z” = BLSTM(h1, θ)

[0023] where BLSTM is the bidirectional long short-term memory network and θ is the parameter of BLSTM;

[0024] S15. Use MSE as the loss function L EMOS , and calculate the loss function L using the measured value z” and the true value z', EMOS to complete the pre-training of the English speech quality assessment model and obtain the trained English speech quality assessment model, where the mathematical expression of L EMOS is:

[0025]

[0026] where n is the number of predicted values.

[0027] Further: The step S2 includes the following sub-steps:

[0028] S21. Initialize the parameters of the Mongolian speech self-supervised model with the parameters learned by the trained English speech self-supervised model, and use the MLM learning task to learn the parameters of the Mongolian speech self-supervised model to complete the transfer learning of the speech self-supervised model and obtain the trained Mongolian speech self-supervised model;

[0029] S22. Initialize the parameters of the Mongolian speech quality assessment model with the parameters learned by the trained English speech quality assessment model, and use the MSE loss function to learn the parameters of the Mongolian speech quality assessment model to complete the transfer learning of the speech quality assessment model and obtain the trained Mongolian speech quality assessment model.

[0030] Further: The step S3 includes the following sub-steps:

[0031] S31. Read the Mongolian speech signal, and use the trained Mongolian speech self-supervised model to extract the feature vectors fs at the frame level from the input Mongolian speech signal, where the expression of fs is:

[0032]

[0033]

[0034] MLP(Z) = ZW + b

[0035]

[0036] Among them, MLP is the feed-forward network layer, is an intermediate variable, MSA is the multi-head attention layer, LN is the normalization layer, x is the Mongolian speech signal, Z is the normalized feature vector, W is the parameter matrix, and b is the bias value; Q, K, and V are query, key, and value respectively, and their values are all x;

[0037] S32. Read the input Mongolian text, and use the pre-trained Mongolian BERT model to extract the text features ft at the utterance level, where the expression of ft is:

[0038] ft = BERT(y, θ BERT )

[0039] Among them, BERT is the BERT model, θ BERT is the parameter of the BERT model, and y is the input Mongolian text.

[0040] Furthermore, the expressions of the semantic feature f and the MOS score z in step S4 are:

[0041] f = conv1(W, (fs + ft)) + b

[0042] z = BLSTM(f, θ)

[0043] The beneficial effects of the present invention are:

[0044] 1. The present invention first creates an automatic evaluation method for Mongolian speech quality, fills the gap in this field, changes the previous way of manually evaluating Mongolian speech quality, and greatly saves the cost of manual evaluation;

[0045] 2. The present invention adopts a hierarchical transfer learning method to fully transfer the knowledge of the English model to the evaluation of Mongolian speech quality, effectively improving the accuracy of Mongolian speech quality evaluation.

[0046] 3. The present invention provides a paradigm for the automatic speech quality assessment of Mongolian, avoiding the time-consuming and laborious manual speech quality evaluation, and effectively improving the problem that the model parameter training is insufficient in the scenario of low-resource training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic flowchart of the Mongolian automatic speech quality assessment method described in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The following describes the specific embodiments of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0049] As Figure 1 shown, in an embodiment of the present invention, a Mongolian automatic speech quality assessment method based on hierarchical transfer learning is provided, including the following steps:

[0050] S1. Pre-train an English speech self-supervised model and an English speech quality assessment model to obtain a trained English speech self-supervised model and a trained English speech quality assessment model;

[0051] S2. Perform transfer learning on the trained English speech self-supervised model and the trained English speech quality assessment model to obtain a trained Mongolian speech self-supervised model and a trained Mongolian speech quality assessment model;

[0052] S3. Use the trained Mongolian speech self-supervised model and the BERT model to extract the feature vector fs of the Mongolian speech and the text feature ft in the corresponding text;

[0053] S4. Fuse fs and ft into a sentence-level semantic feature f, and send f into the trained Mongolian speech quality assessment model to obtain the MOS score z corresponding to the Mongolian speech signal, completing the automatic speech quality assessment of Mongolian.

[0054] In the pre-training stage, a large-scale English speech data is used to train the English speech self-supervised model and the English speech quality assessment model. Since there is a large amount of speech data in English data, the pre-trained model can well learn a high-quality model with certain semantic knowledge.

[0055] In this embodiment, the step S1 includes the following sub-steps:

[0056] S11. Read the English speech signal and mask the Mel spectral features f of 15% of the speech frames mel , which serves as the input sequence xe for the English speech self-supervised model;

[0057] S12. Read the input sequence xe through a deep bidirectional Transformer encoder, calculate the loss function L MLM , and perform backpropagation on it to complete the MLM task and obtain a trained English speech self-supervised model, where the mathematical expression of the loss function L MLM is:

[0058]

[0059] where θ are the bidirectional Transformer parameters, θ1 are the parameters of the output layer connected by the bidirectional Transformer in the MLM task, M represents the set of masked words, p(.) represents the conditional probability, m represents the current frame, mi i represents the i-th speech frame, and V is the number of speech frames;

[0060] S13. Input the Mel spectral features f of the English speech signal mel into the CNN layer, and process f mel through the CNN layer to obtain the local and global information h1 of f mel , where the mathematical expression of h1 is:

[0061] h1 = conv1(W, f mel ) + b

[0062] where conv1(,) is the convolution operation, W is the weight matrix, and b is the bias value;

[0063] S14. Input the local and global information h1 into the bidirectional long short-term memory network to obtain the true value z' of MOS, and calculate the predicted value z” of the MOS score, where the expression of z” is:

[0064] z” = BLSTM(h1, θ)

[0065] where BLSTM is the bidirectional long short-term memory network and θ are the parameters of BLSTM;

[0066] S15. Use MSE as the loss function L EMOS , and calculate the loss function L using the measured value z” and the true value z' EMOS to complete the pre-training of the English speech quality assessment model and obtain a trained English speech quality assessment model, where the mathematical expression of L EMOS is:

[0067]

[0068] Among them, n is the number of predicted values.

[0069] In the transfer learning stage, since the model updates its parameters starting from the parameters learned from large-scale English data, the model can learn robust model parameters with a small amount of Mongolian speech data. After the self-supervised transfer learning of Mongolian speech is completed, the trained self-supervised speech model can be used to extract high-quality feature representations for Mongolian speech, which are used for the evaluation of Mongolian speech quality.

[0070] In this embodiment, step S2 includes the following sub-steps:

[0071] S21. Initialize the parameters of the Mongolian speech self-supervised model with the parameters learned by the trained English speech self-supervised model, and use the MLM learning task to learn the parameters of the Mongolian speech self-supervised model, complete the transfer learning of the self-supervised speech model, and obtain the trained Mongolian speech self-supervised model;

[0072] S22. Initialize the parameters of the Mongolian speech quality assessment model with the parameters learned by the trained English speech quality assessment model, and use the MSE loss function to learn the parameters of the Mongolian speech quality assessment model to complete the transfer learning of the speech quality assessment model, and obtain the trained Mongolian speech quality assessment model.

[0073] In this embodiment, step S3 includes the following sub-steps:

[0074] S31. Read the Mongolian speech signal, and use the trained Mongolian speech self-supervised model to extract feature vectors at the frame level fs from the input Mongolian speech signal. Among them, the expression of fs is:

[0075]

[0076]

[0077] MLP(Z) = ZW + b

[0078]

[0079] Among them, MLP is the feedforward network layer, is the intermediate variable, MSA is the multi-head attention layer, LN is the normalization layer, x is the Mongolian speech signal, Z is the normalized feature vector, W is the parameter matrix, and b is the bias value; Q, K, and V are query, key, and value respectively, and their values are all x;

[0080] S32. Read Mongolian text input and use the pre-trained Mongolian BERT model to propose discourse-level text features ft, where the expression of ft is:

[0081] ft=BERT(y,θ BERT )

[0082] Among them, BERT is the BERT model, θ BERT are the parameters of the BERT model, and y is the Mongolian text input.

[0083] The training method of the BERT model includes the following steps:

[0084] B1. Segment the unlabeled Mongolian text by suffix, and process the segmented text data to obtain the final training data;

[0085] B2. Take the final training data as the input sequence S m , and read S through a deep bidirectional encoder m , calculate the loss function L of the MLM task MLM , and for L MLM Perform back propagation;

[0086] Among them, L MLM The expression is:

[0087]

[0088] Among them, θ is the parameter of the encoder in the BERT model, θ1 is the parameter of the output layer connected to the encoder on the MLM task, M represents the set of masked words, p(.) represents the conditional probability, m represents the current frame, and m i represents the i-th speech frame, and V is the number of speech frames.

[0089] B3. Randomly extract text from the final training data with a probability of 50% as the first input segment S1 of the NSP task, and continue to randomly extract text from the final training data with a probability of 50% as the second input segment S2 of the NSP task to determine whether S1 is the next connecting sentence of S2. Design the loss function L of the NSP task NSP , and for L NSP Perform back propagation;

[0090] Among them, L NSP The expression is:

[0091]

[0092] Among them, θ2 is the parameter of the output layer of the encoder on the NSP task, N is the set of extracted text fragments, n is the current fragment, and n jIt is an element in the set [IsNext,NotNext], IsNext means the next sentence is a connecting sentence, NotNext means the next sentence is not a connecting sentence.

[0093] B4. According to L MLM and L NSP Calculate the BERT model joint learning loss function L BERT , complete the pre-training of the BERT model and obtain the trained BERT model, L BERT The expression is:

[0094] L BERT (θ,θ1,θ2)=L MLM (θ,θ1)+L NSP (θ,θ2)

[0095] In this embodiment, the expressions of the semantic feature f and the MOS score z in step S4 are:

[0096] f=conv1(W,(fs+ft))+b

[0097] z=BLSTM(f,θ)

[0098] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0099] The present invention provides a Mongolian automatic speech quality assessment method based on hierarchical transfer learning, which is the first Mongolian automatic speech quality assessment method, filling the gap in this field, changing the previous manual evaluation method of Mongolian speech quality, and greatly saving the manual evaluation cost.

Claims

1. An automatic Mongolian speech quality assessment method based on hierarchical transfer learning, characterized in that, It includes the following steps: S1. Pre-train the English speech self-supervised model and the English speech quality assessment model to obtain the trained English speech self-supervised model and the trained English speech quality assessment model; S2. Conduct transfer learning on the trained English speech self-supervised model and the trained English speech quality assessment model to obtain the trained Mongolian speech self-supervised model and the trained Mongolian speech quality assessment model; S3. Use the trained Mongolian speech self-supervised model and the BERT model to extract the feature vector fs of the Mongolian speech and the text feature ft in the corresponding text respectively; S4. Fuse fs and ft into the semantic feature f at the sentence level, and send f into the trained Mongolian speech quality assessment model to obtain the MOS score z corresponding to the Mongolian speech signal, thus completing the automatic Mongolian speech quality assessment.

2. The method for automatically evaluating the quality of Mongolian speech based on hierarchical transfer learning according to claim 1, wherein The step S1 includes the following sub-steps: S11. Read the English speech signal and mask the Mel spectral features f of 15% of the speech frames of the signal mel as the input sequence xe of the English speech self-supervised model; S12. Read the input sequence xe through a deep bidirectional Transformer encoder, calculate the loss function L MLM , and perform backpropagation on it to complete the MLM task and obtain a trained English speech self-supervised model, where the loss function L MLM has the following mathematical expression: Among them, θ is the parameter of the bidirectional Transformer, θ1 is the parameter of the output layer connected by the bidirectional Transformer in the MLM task, M represents the set of masked words, p(.) represents the conditional probability, m represents the current frame, and m i represents the i-th speech frame, and V is the number of speech frames; S13. Input the Mel-spectrum features f of the English speech signal mel into the CNN layer and process f through the CNN layer mel to obtain the local and global information h1 of f, where the mathematical expression of h1 is: mel ​ h1 = conv1(W, f mel ) + b Where conv1(,) is the convolution operation, W is the weight matrix, and b is the bias value; S14. Input the local and global information h1 into the bidirectional long short-term memory network to obtain the true value z' of MOS, and calculate the predicted value z” of the MOS score, where the expression of z” is: z”=BLSTM(h1,θ) Where BLSTM is the bidirectional long short-term memory network and θ is the parameter of BLSTM; S15. Use MSE as the loss function L EMOS , and calculate the loss function L using the measured value z” and the true value z' EMOS , complete the pre-training of the English speech quality assessment model, and obtain the trained English speech quality assessment model, where the mathematical expression of L EMOS is: Where n is the number of predicted values.

3. The method for automatically evaluating the speech quality of Mongolian based on hierarchical transfer learning according to claim 2, characterized in that The step S2 includes the following sub-steps: S21. Initialize the parameters of the Mongolian speech self-supervised model with the parameters learned by the trained English speech self-supervised model, and use the MLM learning task to learn the parameters of the Mongolian speech self-supervised model to complete the transfer learning of the speech self-supervised model and obtain the trained Mongolian speech self-supervised model; S22. Initialize the parameters of the Mongolian speech quality assessment model with the parameters learned by the trained English speech quality assessment model, and use the MSE loss function to learn the parameters of the Mongolian speech quality assessment model to complete the transfer learning of the speech quality assessment model and obtain the trained Mongolian speech quality assessment model.

4. The method for automatically evaluating the quality of Mongolian speech based on hierarchical transfer learning according to claim 3, wherein The step S3 includes the following sub-steps: S31. Read the Mongolian speech signal, and use the trained Mongolian speech self-supervised model to extract the frame-level feature vector fs from the input Mongolian speech signal. Where the expression of fs is: MLP(Z)=ZW+b Among them, MLP is a feed-forward network layer, is an intermediate variable, MSA is a multi-head attention layer, LN is a normalization layer, x is a Mongolian speech signal, Z is a normalized feature vector, W is a parameter matrix, and b is a bias value; Q, K, and V are query, key, and value respectively, and their values are all x; S32. Read the input of the Mongolian text, and use the pre-trained Mongolian BERT model to extract the discourse-level text feature ft. Where the expression of ft is: ft = BERT(y, θ BERT ) Among them, BERT is the BERT model, and θ BERT are the parameters of the BERT model, and y is the Mongolian text input.

5. The method for automatically evaluating the quality of Mongolian speech based on hierarchical transfer learning according to claim 4, wherein The expressions of the semantic feature f and the MOS score z in the step S4 are: f=conv1(W,(fs+ft))+b z=BLSTM(f,θ)

Images