Training method, device and storage medium for speech quality evaluation model

Abstract

The present application relates to the field of artificial intelligence technology. The present application provides a training method, device and storage medium for a speech quality evaluation model. The training method for the speech quality evaluation model includes: obtaining a sample speech signal; extracting speech features from the sample speech signal to obtain speech features of the sample speech signal; obtaining a predicted quality score of the sample speech signal based on speech features through a quality score evaluation network; obtaining a predicted quality classification of the sample speech signal based on speech features through a first classification network; obtaining a predicted source classification of the sample speech signal based on speech features through a second classification network; calculating the total loss function value of the speech quality evaluation model based on the predicted quality score, predicted quality classification and predicted source classification, and adjusting the parameters of the speech quality evaluation model according to the total loss function value. Implementing the present application can improve the accuracy of the model in evaluating speech quality.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device and storage medium for a speech quality evaluation model. Background Art

[0002] With the development of speech synthesis technology, machine-synthesized speech has been applied to various scenarios in life, such as high-speed rail broadcasts, customer service, sales, etc., realizing the application of speech synthesis technology. The quality of synthesized speech, that is, naturalness, is the key to the speech synthesis system. However, due to the high subjectivity of speech quality evaluation tasks, the quality of synthesized speech is still evaluated by subjective Mean Opinion Score (MOS) tests. The MOS test requires multiple experts or trained scorers to score the speech, which is a time-consuming and labor-intensive process. In order to achieve automated synthetic speech quality evaluation, some researchers proposed MOSnet, which predicts the MOS score of a piece of speech through an end-to-end neural network model. During the model training process, the model parameters are adjusted by the difference between the predicted MOS score and the actual score of the speech. This model training method is based on model training for a single task, and the trained model is inaccurate in speech quality evaluation. Summary of the invention

[0003] Based on this, it is necessary to provide a training method, device and storage medium for a speech quality evaluation model to address the above-mentioned technical problems, which can enable the model to pay more attention to high-quality synthesized speech and improve the model's ability to extract low-level features, thereby improving the accuracy of the speech quality evaluation model in evaluating speech quality.

[0004] In a first aspect, the present application provides a method for training a speech quality assessment model, wherein the speech quality assessment model includes a speech feature extraction network, a quality score assessment network, a first classification network, and a second classification network; the method includes:

[0005] Acquire a sample speech signal;

[0006] Extracting speech features from the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal;

[0007] Obtaining a predicted quality score of the sample speech signal based on the speech feature through the quality score evaluation network;

[0008] Obtaining, by the first classification network, a prediction quality classification of the sample speech signal based on the speech feature, the prediction quality classification being used to indicate whether the speech signal is predicted to be a synthetic human voice or a natural human voice;

[0009] Obtaining a predicted source classification of the sample speech signal based on the speech feature through the second classification network, wherein the predicted source classification is used to indicate a predicted source object of the speech signal;

[0010] Based on the predicted quality score, the predicted quality classification and the predicted source classification, a total loss function value of the speech quality assessment model is calculated, and the parameters of the speech quality assessment model are adjusted according to the total loss function value.

[0011] In combination with the first aspect, in some embodiments, the calculating the total loss function value of the speech quality evaluation model based on the predicted quality score, the predicted quality classification, and the predicted source classification includes:

[0012] Calculating a first loss function value based on the predicted quality score and the actual quality score of the sample speech signal;

[0013] Calculating a second loss function value based on the predicted quality classification and the actual quality classification of the sample speech signal;

[0014] Calculating a third loss function value based on the predicted source classification and the actual source classification of the sample speech signal;

[0015] The first loss function value, the second loss function value and the third loss function value are weightedly summed to obtain a total loss function value of the speech quality evaluation model.

[0016] In combination with the first aspect, in some embodiments, before extracting speech features from the sample speech signal through the speech feature extraction network to obtain the speech features of the sample speech signal, the method further includes:

[0017] Dividing the sample speech signal into a plurality of speech frames;

[0018] The extracting speech features of the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal includes:

[0019] Each speech frame among the multiple speech frames is input into the speech feature extraction network, and the speech features of each speech frame are extracted through the speech feature extraction network.

[0020] In combination with the first aspect, in some embodiments, the quality score evaluation network includes a first fully connected layer and a first pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0021] The obtaining, by the quality score evaluation network, a predicted quality score of the speech signal based on the speech feature comprises:

[0022] Inputting a speech feature of each speech frame of the plurality of speech frames into the first fully connected layer, processing the speech feature of each speech frame through the first fully connected layer, and obtaining a first prediction quality score corresponding to each speech frame;

[0023] Inputting the first prediction quality score corresponding to each speech frame and the sentence label marked by each speech frame into the first pooling layer, and performing average pooling processing on the first prediction quality scores corresponding to the speech frames with the same sentence label in the multiple speech frames through the first pooling layer to obtain a second prediction quality score corresponding to each sentence in at least one sentence associated with the multiple speech frames;

[0024] The first prediction quality score corresponding to each speech frame and the second prediction quality score corresponding to each sentence are determined as the prediction quality score of the sample speech signal.

[0025] In combination with the first aspect, in some embodiments, calculating the first loss function value based on the predicted quality score and the actual quality score of the sample speech signal includes:

[0026] Determine a loss function value corresponding to each speech frame based on a first predicted quality score corresponding to each speech frame and a first actual quality score corresponding to each speech frame;

[0027] Determining a loss function value corresponding to each sentence based on a second predicted quality score corresponding to each sentence and a second actual quality score corresponding to each sentence;

[0028] The loss function value corresponding to the speech frame and the loss function value corresponding to the sentence are weightedly summed to obtain a first loss function value.

[0029] In combination with the first aspect, in some embodiments, the first classification network includes a second fully connected layer and a second pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0030] Obtaining a predicted quality classification of the sample speech signal based on the speech feature through the first classification network includes:

[0031] Inputting the speech feature of each speech frame of the plurality of speech frames into the second fully connected layer, processing the speech feature of each speech frame through the second fully connected layer, and obtaining first probability information corresponding to each speech frame, wherein the first probability information is used to indicate the probability that the speech frame belongs to a synthetic human voice and a natural human voice;

[0032] The first probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the second pooling layer, and the first probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the second pooling layer to obtain the prediction quality classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the prediction quality classification corresponding to each sentence is determined as the prediction quality classification of the sample speech signal.

[0033] In combination with the first aspect, in some embodiments, the second classification network includes a third fully connected layer and a third pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0034] The step of obtaining the predicted source classification of the sample speech signal based on the speech feature by the second classification network includes:

[0035] Inputting the speech feature of each speech frame of the multiple speech frames into the third fully connected layer, processing the speech feature of each speech frame through the third fully connected layer, and obtaining second probability information corresponding to each speech frame, wherein the second probability information is used to indicate the probability that the speech frame is derived from M source objects, where M is an integer greater than or equal to 2;

[0036] The second probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the third pooling layer, and the second probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the third pooling layer to obtain the predicted source classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the predicted source classification corresponding to each sentence is determined as the predicted source classification of the sample speech signal.

[0037] In a second aspect, the present application provides a training device for a speech quality evaluation model, wherein the speech quality evaluation model includes a speech feature extraction network, a quality score evaluation network, a first classification network, and a second classification network; the device includes:

[0038] A speech signal acquisition unit, used to acquire a sample speech signal;

[0039] A feature extraction unit, configured to extract speech features from the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal;

[0040] A quality score acquisition unit, configured to obtain a predicted quality score of the sample speech signal based on the speech feature through the quality score evaluation network;

[0041] A first classification unit, configured to obtain a prediction quality classification of the sample speech signal based on the speech feature through the first classification network, wherein the prediction quality classification is used to indicate whether the speech signal is predicted to be a synthetic human voice or a natural human voice;

[0042] A second classification unit, configured to obtain a predicted source classification of the sample speech signal based on the speech feature through the second classification network, wherein the predicted source classification is used to indicate a source object of the predicted speech signal;

[0043] A parameter adjustment unit is used to calculate the total loss function value of the speech quality evaluation model based on the prediction quality score, the prediction quality classification and the prediction source classification, and adjust the parameters of the speech quality evaluation model according to the total loss function value.

[0044] In conjunction with the second aspect, in some embodiments, the parameter adjustment unit is specifically used to:

[0045] Calculating a first loss function value based on the predicted quality score and the actual quality score of the sample speech signal;

[0046] Calculating a second loss function value based on the predicted quality classification and the actual quality classification of the sample speech signal;

[0047] Calculating a third loss function value based on the predicted source classification and the actual source classification of the sample speech signal;

[0048] The first loss function value, the second loss function value and the third loss function value are weightedly summed to obtain a total loss function value of the speech quality evaluation model.

[0049] In conjunction with the second aspect, in some embodiments, the device further includes:

[0050] A division unit, used for dividing the sample speech signal into a plurality of speech frames;

[0051] The feature extraction unit is specifically used to: input each speech frame of the multiple speech frames into the speech feature extraction network, and extract the speech features of each speech frame through the speech feature extraction network.

[0052] In conjunction with the second aspect, in some embodiments, the quality score evaluation network includes a first fully connected layer and a first pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0053] The quality score acquisition unit is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the first fully connected layer, process the speech feature of each speech frame through the first fully connected layer, and obtain a first predicted quality score corresponding to each speech frame;

[0054] Inputting the first prediction quality score corresponding to each speech frame and the sentence label marked by each speech frame into the first pooling layer, and performing average pooling processing on the first prediction quality scores corresponding to the speech frames with the same sentence label in the multiple speech frames through the first pooling layer to obtain a second prediction quality score corresponding to each sentence in at least one sentence associated with the multiple speech frames;

[0055] The first prediction quality score corresponding to each speech frame and the second prediction quality score corresponding to each sentence are determined as the prediction quality score of the sample speech signal.

[0056] In combination with the second aspect, in some embodiments, the parameter adjustment unit is specifically used to: determine the loss function value corresponding to the speech frame based on the first predicted quality score corresponding to each speech frame and the first actual quality score corresponding to each speech frame;

[0057] Determining a loss function value corresponding to each sentence based on a second predicted quality score corresponding to each sentence and a second actual quality score corresponding to each sentence;

[0058] The loss function value corresponding to the speech frame and the loss function value corresponding to the sentence are weightedly summed to obtain a first loss function value.

[0059] In conjunction with the second aspect, in some embodiments, the first classification network includes a second fully connected layer and a second pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0060] The first classification unit is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the second fully connected layer, process the speech feature of each speech frame through the second fully connected layer, and obtain first probability information corresponding to each speech frame, wherein the first probability information is used to indicate the probability that the speech frame belongs to a synthetic human voice and a natural human voice;

[0061] The first probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the second pooling layer, and the first probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the second pooling layer to obtain the prediction quality classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the prediction quality classification corresponding to each sentence is determined as the prediction quality classification of the sample speech signal.

[0062] In combination with the second aspect, in some embodiments, the second classification network includes a third fully connected layer and a third pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame; the second classification unit is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the third fully connected layer, and process the speech feature of each speech frame through the third fully connected layer to obtain second probability information corresponding to each speech frame, and the second probability information is used to indicate the probability that the speech frame is derived from M source objects, and M is an integer greater than or equal to 2;

[0063] The second probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the third pooling layer, and the second probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the third pooling layer to obtain the predicted source classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the predicted source classification corresponding to each sentence is determined as the predicted source classification of the sample speech signal.

[0064] In a third aspect, the present application provides a training device for a speech quality assessment model, comprising a processor, a memory, and a communication interface, wherein the processor, the memory, and the communication interface are interconnected, wherein the communication interface is used to receive and send data, the memory is used to store program code, and the processor is used to call the program code to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program; when the computer program runs on one or more processors, the terminal device executes the method described in the first aspect and any possible implementation of the first aspect.

[0066] In an embodiment of the present application, when training a speech quality evaluation model, two auxiliary tasks are introduced to train the model. One auxiliary task is to predict whether the sample speech signal is a synthetic human voice or a natural human voice based on the speech features of the sample speech signal through a first classification network, and the other auxiliary task is to predict the predicted source classification of the sample speech signal based on the speech features of the sample speech signal through a second classification network. The introduced auxiliary task of judging whether the sample speech signal is a natural human voice can make the model pay more attention to high-quality synthesized speech, and to a certain extent alleviate the problem of the imbalance in the number of sample speech signals of synthetic human voices and natural human voices in the training set. The introduced auxiliary task of judging the source of the sample speech signal can enhance the model's ability to extract low-level features. The addition of these two auxiliary tasks improves the performance of the model, especially the accuracy of the model's evaluation of high-quality synthesized speech, and can make the quality score predicted by the model more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0068] Figure 1 A flowchart of a method for training a speech quality assessment model provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of a voice quality evaluation model provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of a training device for a speech quality assessment model provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of another training device for a speech quality assessment model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0073] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and appended claims of the present application, the singular expressions "a", "a", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context.

[0074] In the present application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".

[0075] The embodiments of the present application can acquire and process relevant voice data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] An embodiment of the present application provides a method for training a speech quality assessment model. In order to more clearly describe the solution of the present application, some drawings involved in the present application are further introduced below.

[0078] See also Figure 1 , Figure 1 A flow chart of a method for training a speech quality evaluation model provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0079] Step 110, obtaining a sample speech signal;

[0080] Step 120, extracting speech features from the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal;

[0081] In the embodiment of the present application, the sample speech signal may be a speech signal in a training set, which includes various synthetic human voice speech signals and natural human voice speech signals. Of course, the speech signals in the training set may come from different source objects, for example, they may include speech signals from robot A, robot B, and may also include speech signals from natural person A and natural person B. Among them, each sample speech signal in the training set is used to train the speech quality evaluation model of the present application, and the speech quality evaluation model of the present application may include a speech feature extraction network, a quality score evaluation network (i.e., the main task), a first classification network (i.e., auxiliary task 1) and a second classification network (i.e., auxiliary task 2), wherein the speech feature extraction network is respectively connected to the quality score evaluation network, the first classification network and the second classification network, and the speech features of the sample speech signal are extracted through the speech feature extraction network, and the extracted speech features are respectively input into the quality score evaluation network, the first classification network and the second classification network for processing.

[0082] Specific options, such as Figure 2 As shown, the sample speech signal is first input into the spectrum input layer for spectrum sequence conversion. The spectrum input layer converts the sample speech signal into a spectrum sequence through short-time Fourier transform (STFT), and divides the spectrum sequence into multiple speech frames, each of which has a duration of 32ms and a frame displacement of 16ms. Optionally, the sample speech signal may be speech data including multiple sentences. After the sample speech signal is divided into multiple speech frames, a sentence label may be annotated for each speech frame according to the sentence associated with each speech frame, and the sentence labels annotated for the speech frames associated with the same sentence are the same.

[0083] Furthermore, the spectrum input layer inputs the spectrum sequence of each speech frame into the speech feature extraction network to extract speech features, thereby obtaining the speech features of each speech frame. Figure 2 As shown, the speech feature extraction network can be a CNN-LSTM feature extraction layer, which is composed of a CNN feature extraction unit (i.e., a convolution Conv layer) and a bidirectional LSTM unit. The CNN feature extraction unit can extract the spatial features of the speech frame level of the spectrum sequence. In the model, each CNN feature extraction unit is composed of 3 layers of CNN in series, and there can be multiple CNN feature extraction units in the model. The bidirectional LSTM unit extracts the forward time series features and backward time series features of the spectrum sequence, and can capture the long-term features and short-term features of the spectrum sequence at the same time. The extracted speech features at the speech frame level are then reduced in dimension through a fully connected layer (i.e., FC) to obtain the reduced-dimensional speech features, which are then input into the multi-task output layer, as shown in FIG. Figure 2As shown, the multi-task output layer includes three branches, namely, the output layer corresponding to the main task and the output layers corresponding to the two auxiliary tasks. The two auxiliary tasks are referred to as auxiliary task 1 and auxiliary task 2 in this application. Auxiliary task 1 is used to perform binary classification of natural human voice or synthetic human voice on the sample speech signal, and auxiliary task 2 is used to perform multi-classification on the source object of the sample speech signal. In the embodiment of the present application, the main task output layer is called the quality score average network, the auxiliary task 1 output layer is called the first classification network, and the auxiliary task 2 output layer is called the second classification network.

[0084] Step 130, obtaining a predicted quality score of the sample speech signal based on the speech feature through the quality score evaluation network;

[0085] Step 140, obtaining a predicted quality classification of the sample speech signal based on the speech feature through the first classification network, wherein the predicted quality classification is used to indicate whether the sample speech signal is a synthetic human voice or a natural human voice;

[0086] Step 150, obtaining a predicted source classification of the sample speech signal based on the speech feature through the second classification network, wherein the predicted source classification is used to represent a source object of the sample speech signal;

[0087] Step 160, based on the predicted quality score, the predicted quality classification and the predicted source classification, calculate the total loss function value of the speech quality evaluation model, and adjust the parameters of the speech quality evaluation model according to the total loss function value.

[0088] In the embodiment of the present application, the speech features of each speech frame of the extracted multiple speech frames are respectively input into the quality score evaluation network, the first classification network and the second classification network.

[0089] Exemplarily, the quality score evaluation network (i.e., the main task output layer) includes a first fully connected layer, which obtains a predicted quality score of each speech frame based on the speech features of each speech frame, such as a predicted MOS score of each speech frame. Further optionally, the quality score evaluation network may also include a first pooling layer, which inputs the predicted quality score of each speech frame and the sentence label annotated with each speech frame into the first pooling layer, and performs average pooling processing on the predicted quality scores of speech frames with the same sentence label through the first pooling layer to obtain a predicted quality score for each sentence in at least one sentence associated with multiple speech frames. The predicted quality scores at the frame level are aggregated into predicted quality scores at the sentence level through average pooling processing.

[0090] In calculating the first loss function value L of the main task mainWhen , the first loss function can be used for calculation, for example, the MSE function is used to measure the size of the error. In the following formula, the first term represents the MOS score error at the sentence level, that is, the loss function value corresponding to the sentence, which is obtained based on the difference between the predicted quality score of each sentence in the sample speech signal and the actual quality score corresponding to each sentence. The second term represents the MOS score error at the speech frame level, that is, the loss function value corresponding to the speech frame, which is obtained based on the difference between the predicted quality score of each speech frame in the sample speech signal and the actual quality score of each speech frame:

[0091]

[0092] Score: Q u represents the predicted MOS score of the uth sentence; T u Indicates the length of the u-th sentence (i.e., how many speech frames a sentence has); α f is a parameter that controls the weight; q u, represents the predicted MOS score of the tth frame of the uth sentence.

[0093] Exemplarily, the first classification network (i.e., the output layer of auxiliary task 1) may include a second fully connected layer and a softmax classifier, which may obtain a predicted quality classification of the sample speech signal based on speech features at the speech frame level, wherein the predicted quality classification is natural voice or synthetic voice. A second loss function is used to calculate a second loss function value corresponding to the first classification network according to the predicted quality classification of the sample speech signal and the actual quality classification of the sample speech signal.

[0094] Specific options, such as Figure 2As shown, the output layer of auxiliary task 1 is to determine whether the sample speech signal is a natural human voice or a synthetic human voice, that is, a binary classification problem. In the general large-scale speech training set, there are a large number of low-quality synthetic human voices, while the quality of natural human voices is relatively high. There are relatively few natural human voice signals in the training set. If only the sample speech signals in the training set are used to train the output layer corresponding to the main task, this uneven distribution of data quality makes the trained model perform poorly. Generally speaking, it is more difficult to distinguish whether a high-quality speech signal is a synthetic speech. Therefore, introducing the auxiliary task 1 during model training can make the model pay more attention to high-quality synthetic speech, which to a certain extent alleviates the problem of uneven quality of sample speech signals in the data set and improves the performance of the model. During training, the speech features of each speech frame are passed through the second fully connected layer in the output layer corresponding to the auxiliary task 1 to obtain the first probability information corresponding to each speech frame, that is, a two-dimensional vector. The first probability information is used to represent the probability that the speech signal of the speech frame is a synthetic human voice and a natural human voice. Optionally, the output layer of auxiliary task 1 may further include a second pooling layer, and the two-dimensional vector corresponding to each speech frame and the sentence label annotated by each speech frame may be input into the second pooling layer, and the two-dimensional vectors of the speech frames with the same sentence label are further processed by average pooling through the second pooling layer to obtain the probability that the sentence associated with each speech frame is a synthetic voice and a natural voice, so that the classification with the maximum probability is used as the sentence prediction quality classification, that is, predicting that each sentence is a natural voice or a synthetic voice. When calculating the second loss function value of auxiliary task 1, the second loss function may use a cross entropy function, as shown in the following formula:

[0095]

[0096] The predicted probability of the i-th class (between 0 and 1).

[0097] Exemplarily, the second classification network (i.e., the output layer of auxiliary task 2) includes a third fully connected layer and a softmax classifier. The third fully connected layer and the softmax classifier can obtain a predicted source classification of the sample speech signal based on speech features. The predicted source classification is used to represent the source object of the sample speech signal. The second classification network can be a multi-classification network. For example, M source objects are preset, where M is an integer greater than or equal to 2. Then, it can be predicted that the sample speech signal is derived from one of the M source objects.

[0098] Specific options, such as Figure 2As shown, the output layer of auxiliary task 2 is to determine the source object of the speech signal, that is, a multi-classification problem, for example, to determine whether the sample speech signal comes from robot A or robot B. The source of the speech signal determines the low-level features of the speech, such as acoustic rhythm. By introducing this auxiliary task 2, the model's ability to extract low-level features can be enhanced, which ultimately improves the model's performance in speech quality evaluation. During training, the speech features of each speech frame are processed by the third fully connected layer in the output layer corresponding to auxiliary task 2 to obtain the second probability information corresponding to each speech frame, that is, to obtain an M-dimensional vector, where M is the number of preset source objects (i.e., the sum of the number of human voice sources and the number of synthetic sources). The M-dimensional vector is used to represent the probability that the speech frame comes from each source object among the M source objects. Optionally, the output layer of auxiliary task 2 may further include a third pooling layer, and the second probability information corresponding to each speech frame and the sentence label annotated by each speech frame are input into the third pooling layer, and the M-dimensional feature vectors corresponding to the speech frames with the same sentence label are averagely pooled through the third pooling layer to obtain the probability that the sentences associated with each speech frame are derived from M source objects, thereby determining the source object with the highest probability as the predicted source classification of the sentence. When calculating the third loss function value of auxiliary task 2, the loss function may also use a cross entropy function, as shown in the following formula:

[0099]

[0100] Where U is the number of sentences in the input speech signal; represents the actual probability that the uth sentence belongs to the i-th category (the probability that the sentence belongs to the i-th category is 1, does not belong to the i-th category =0); C u, represents the predicted probability (between 0 and 1) that the uth sentence belongs to the ith category.

[0101] Furthermore, the first loss function value, the second loss function value and the third loss function value are weightedly summed to obtain the total loss function value, and the parameters of the speech quality evaluation model are adjusted based on the total loss function value, that is, the parameters of the CNN-LSTM feature extraction layer, the first fully connected layer, the second fully connected layer and the third fully connected layer are adjusted. The training is continuously iterated until the model converges to obtain the trained speech quality evaluation model.

[0102] As shown in the following formula, it is the formula for calculating the total loss function value L, where α 0 , α 1 , α 2 is the parameter that controls the loss ratio:

[0103] L=α 0 L main +α 1 Laux1 +α 2 L aux2 .

[0104] When using the trained speech quality evaluation model to perform quality scoring on the synthetic speech signal to be detected, the synthetic speech signal to be detected is first input into the spectrum input layer, the CNN-LSTM feature extraction layer and the first fully connected layer to obtain the frame-level MOS score. Optionally, the frame-level MOS score is average-pooled to obtain the sentence-level MOS score.

[0105] In an embodiment of the present application, when training a speech quality evaluation model, two auxiliary tasks are introduced to train the model. One auxiliary task is to predict whether the sample speech signal is a synthetic human voice or a natural human voice based on the speech features of the sample speech signal through a first classification network, and the other auxiliary task is to predict the predicted source classification of the sample speech signal based on the speech features of the sample speech signal through a second classification network. The introduced auxiliary task of judging whether the sample speech signal is a natural human voice can make the model pay more attention to high-quality synthesized speech, and to a certain extent alleviate the problem of the imbalance in the number of sample speech signals of synthetic human voices and natural human voices in the training set. The introduced auxiliary task of judging the source of the sample speech signal can enhance the model's ability to extract low-level features. The addition of these two auxiliary tasks improves the performance of the model, especially the accuracy of the model's evaluation of high-quality synthesized speech, and can make the quality score predicted by the model more accurate.

[0106] See also Figure 3 , is a structural diagram of a training device for a speech quality evaluation model provided in an embodiment of the present application, wherein the speech quality evaluation model of the present application includes a speech feature extraction network, a quality score evaluation network, a first classification network, and a second classification network. Figure 3 As shown, the training device of the speech quality evaluation model may include:

[0107] A speech signal acquisition unit 10, used to acquire a sample speech signal;

[0108] A feature extraction unit 11 is used to extract speech features from the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal;

[0109] A quality score acquisition unit 12, configured to obtain a predicted quality score of the sample speech signal based on the speech feature through the quality score evaluation network;

[0110] A first classification unit 13, configured to obtain a prediction quality classification of the sample speech signal based on the speech feature through the first classification network, wherein the prediction quality classification is used to indicate whether the speech signal is predicted to be a synthetic human voice or a natural human voice;

[0111] A second classification unit 14, configured to obtain a predicted source classification of the sample speech signal based on the speech feature through the second classification network, wherein the predicted source classification is used to indicate a source object of the predicted speech signal;

[0112] The parameter adjustment unit 15 is used to calculate the total loss function value of the speech quality evaluation model based on the prediction quality score, the prediction quality classification and the prediction source classification, and adjust the parameters of the speech quality evaluation model according to the total loss function value.

[0113] In a possible design, the parameter adjustment unit 15 is specifically used for:

[0114] Calculating a first loss function value based on the predicted quality score and the actual quality score of the sample speech signal;

[0115] Calculating a second loss function value based on the predicted quality classification and the actual quality classification of the sample speech signal;

[0116] Calculating a third loss function value based on the predicted source classification and the actual source classification of the sample speech signal;

[0117] The first loss function value, the second loss function value and the third loss function value are weightedly summed to obtain a total loss function value of the speech quality evaluation model.

[0118] In one possible design, the device further includes:

[0119] A division unit, used for dividing the sample speech signal into a plurality of speech frames;

[0120] The feature extraction unit is specifically used to: input each speech frame of the multiple speech frames into the speech feature extraction network, and extract the speech features of each speech frame through the speech feature extraction network.

[0121] In a possible design, the quality score evaluation network includes a first fully connected layer and a first pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0122] The quality score acquisition unit 12 is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the first fully connected layer, process the speech feature of each speech frame through the first fully connected layer, and obtain a first predicted quality score corresponding to each speech frame;

[0123] Inputting the first prediction quality score corresponding to each speech frame and the sentence label marked by each speech frame into the first pooling layer, and performing average pooling processing on the first prediction quality scores corresponding to the speech frames with the same sentence label in the multiple speech frames through the first pooling layer to obtain a second prediction quality score corresponding to each sentence in at least one sentence associated with the multiple speech frames;

[0124] The first prediction quality score corresponding to each speech frame and the second prediction quality score corresponding to each sentence are determined as the prediction quality score of the sample speech signal.

[0125] In a possible design, the parameter adjustment unit 15 is specifically used to: determine the loss function value corresponding to the speech frame based on the first predicted quality score corresponding to each speech frame and the first actual quality score corresponding to each speech frame;

[0126] Determining a loss function value corresponding to each sentence based on a second predicted quality score corresponding to each sentence and a second actual quality score corresponding to each sentence;

[0127] The loss function value corresponding to the speech frame and the loss function value corresponding to the sentence are weightedly summed to obtain a first loss function value.

[0128] In a possible design, the first classification network includes a second fully connected layer and a second pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame;

[0129] The first classification unit 13 is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the second fully connected layer, process the speech feature of each speech frame through the second fully connected layer, and obtain first probability information corresponding to each speech frame, wherein the first probability information is used to indicate the probability of the speech frame belonging to a synthetic human voice and a natural human voice;

[0130] The first probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the second pooling layer, and the first probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the second pooling layer to obtain the prediction quality classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the prediction quality classification corresponding to each sentence is determined as the prediction quality classification of the sample speech signal.

[0131] In a possible design, the second classification network includes a third fully connected layer and a third pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame; the second classification unit 14 is specifically used to: input the speech feature of each speech frame of the multiple speech frames into the third fully connected layer, and process the speech feature of each speech frame through the third fully connected layer to obtain the second probability information corresponding to each speech frame, and the second probability information is used to indicate the probability that the speech frame is derived from M source objects, and M is an integer greater than or equal to 2;

[0132] The second probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the third pooling layer, and the second probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the third pooling layer to obtain the predicted source classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the predicted source classification corresponding to each sentence is determined as the predicted source classification of the sample speech signal.

[0133] in, Figure 3 The detailed description of the embodiment of the device shown can refer to the aforementioned Figure 1 The specific description of the illustrated method embodiment will not be repeated here.

[0134] Please refer to Figure 4 , is a structural diagram of another training device for a speech quality evaluation model provided in an embodiment of the present application, such as Figure 4 As shown, the training device 1000 of the speech quality assessment model may include: at least one processor 1001, such as a CPU, at least one communication interface 1003, a memory 1004, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The communication interface 1003 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1004 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1004 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Figure 4 As shown, the memory 1004 as a computer storage medium may include an operating system, a network communication unit, and program instructions.

[0135] exist Figure 4 In the training device 1000 for the speech quality assessment model shown, the processor 1001 can be used to load the program instructions stored in the memory 1004, and specifically perform the following operations:

[0136] Acquire a sample speech signal;

[0137] Extracting speech features from the sample speech signal through the speech feature extraction network to obtain speech features of the sample speech signal;

[0138] Obtaining a predicted quality score of the sample speech signal based on the speech feature through the quality score evaluation network;

[0139] Obtaining, by the first classification network, a prediction quality classification of the sample speech signal based on the speech feature, the prediction quality classification being used to indicate whether the speech signal is predicted to be a synthetic human voice or a natural human voice;

[0140] Obtaining a predicted source classification of the sample speech signal based on the speech feature through the second classification network, wherein the predicted source classification is used to indicate a predicted source object of the speech signal;

[0141] Based on the predicted quality score, the predicted quality classification and the predicted source classification, a total loss function value of the speech quality assessment model is calculated, and the parameters of the speech quality assessment model are adjusted according to the total loss function value.

[0142] It should be noted that the specific implementation process can be found in Figure 1 The specific description of the illustrated method embodiment will not be repeated here.

[0143] The present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figure 1 The specific description of the illustrated embodiment will not be repeated here.

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0145] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A method for training a speech quality assessment model, characterized in that: The speech quality evaluation model includes a speech feature extraction network, a quality score evaluation network, a first classification network and a second classification network; the method includes: Acquire a sample speech signal; Dividing the sample speech signal into a plurality of speech frames, and labeling each speech frame with a sentence label according to the sentence associated with each speech frame, wherein the sentence labels labeled for speech frames associated with the same sentence are the same; Inputting each speech frame of the plurality of speech frames into the speech feature extraction network, and extracting speech features of each speech frame through the speech feature extraction network; Inputting the extracted speech features of each speech frame of the plurality of speech frames into the quality score evaluation network, the first classification network and the second classification network respectively; The quality score evaluation network includes a first fully connected layer and a first pooling layer; Inputting a speech feature of each speech frame of the plurality of speech frames into the first fully connected layer, processing the speech feature of each speech frame through the first fully connected layer, and obtaining a first prediction quality score corresponding to each speech frame; Inputting the first prediction quality score corresponding to each speech frame and the sentence label marked by each speech frame into the first pooling layer, and performing average pooling processing on the first prediction quality scores corresponding to the speech frames with the same sentence label in the multiple speech frames through the first pooling layer to obtain a second prediction quality score corresponding to each sentence in at least one sentence associated with the multiple speech frames; Determine the first prediction quality score corresponding to each speech frame and the second prediction quality score corresponding to each sentence as the prediction quality score of the sample speech signal; Obtaining, by the first classification network, a predicted quality classification of the sample speech signal based on the speech features of each speech frame, the predicted quality classification being used to indicate whether the sample speech signal is a synthetic human voice or a natural human voice; Obtaining, by the second classification network, a predicted source classification of the sample speech signal based on the speech features of each speech frame, the predicted source classification being used to represent a source object of the sample speech signal, the source object of the sample speech signal being one of M preset source objects, where M is an integer greater than or equal to 2; Based on the predicted quality score, the predicted quality classification and the predicted source classification, a total loss function value of the speech quality assessment model is calculated, and parameters of the speech quality assessment model are adjusted according to the total loss function value.

2. The method according to claim 1, characterized in that The calculating the total loss function value of the speech quality evaluation model based on the predicted quality score, the predicted quality classification and the predicted source classification includes: Calculating a first loss function value based on the predicted quality score and the actual quality score of the sample speech signal; Calculating a second loss function value based on the predicted quality classification and the actual quality classification of the sample speech signal; Calculating a third loss function value based on the predicted source classification and the actual source classification of the sample speech signal; The first loss function value, the second loss function value and the third loss function value are weightedly summed to obtain a total loss function value of the speech quality evaluation model.

3. The method according to claim 2, characterized in that The calculating a first loss function value based on the predicted quality score and the actual quality score of the sample speech signal comprises: Determine a loss function value corresponding to each speech frame based on a first predicted quality score corresponding to each speech frame and a first actual quality score corresponding to each speech frame; Determining a loss function value corresponding to each sentence based on a second predicted quality score corresponding to each sentence and a second actual quality score corresponding to each sentence; The loss function value corresponding to the speech frame and the loss function value corresponding to the sentence are weightedly summed to obtain a first loss function value.

4. The method according to claim 1, characterized in that The first classification network includes a second fully connected layer and a second pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame; Obtaining a predicted quality classification of the sample speech signal based on the speech feature through the first classification network includes: Inputting the speech feature of each speech frame of the plurality of speech frames into the second fully connected layer, processing the speech feature of each speech frame through the second fully connected layer, and obtaining first probability information corresponding to each speech frame, wherein the first probability information is used to indicate the probability that the speech frame belongs to a synthetic human voice and a natural human voice; The first probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the second pooling layer, and the first probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the second pooling layer to obtain the prediction quality classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the prediction quality classification corresponding to each sentence is determined as the prediction quality classification of the sample speech signal.

5. The method according to claim 1, characterized in that The second classification network includes a third fully connected layer and a third pooling layer; each speech frame is labeled with a sentence label, and the sentence label is used to indicate the sentence associated with the speech frame; The step of obtaining the predicted source classification of the sample speech signal based on the speech feature by the second classification network includes: Inputting the speech feature of each speech frame of the multiple speech frames into the third fully connected layer, processing the speech feature of each speech frame through the third fully connected layer, and obtaining second probability information corresponding to each speech frame, wherein the second probability information is used to indicate the probability that the speech frame is derived from M source objects, where M is an integer greater than or equal to 2; The second probability information corresponding to each speech frame and the sentence label marked by each speech frame are input into the third pooling layer, and the second probability information corresponding to the speech frames with the same sentence label in the multiple speech frames is averaged pooled through the third pooling layer to obtain the predicted source classification corresponding to each sentence in at least one sentence associated with the multiple speech frames, and the predicted source classification corresponding to each sentence is determined as the predicted source classification of the sample speech signal.

6. A training device for a speech quality evaluation model, characterized in that: The speech quality evaluation model includes a speech feature extraction network, a quality score evaluation network, a first classification network and a second classification network; the device includes: A speech signal acquisition unit, used to acquire a sample speech signal; A division unit, used for dividing the sample speech signal into a plurality of speech frames, and marking a sentence label for each speech frame according to a sentence associated with each speech frame, wherein the sentence labels marked on speech frames associated with the same sentence are the same; A feature extraction unit, configured to input each of the plurality of speech frames into the speech feature extraction network, extract speech features of each of the speech frames through the speech feature extraction network, and input the extracted speech features of each of the plurality of speech frames into the quality score evaluation network, the first classification network, and the second classification network, respectively; the quality score evaluation network comprises a first fully connected layer and a first pooling layer; A quality score acquisition unit, configured to input the speech features of each speech frame of the multiple speech frames into the first fully connected layer, process the speech features of each speech frame through the first fully connected layer, and obtain a first predicted quality score corresponding to each speech frame; input the first predicted quality score corresponding to each speech frame and the sentence label annotated by each speech frame into the first pooling layer, and perform average pooling processing on the first predicted quality scores corresponding to the speech frames with the same sentence label in the multiple speech frames through the first pooling layer to obtain a second predicted quality score corresponding to each sentence in at least one sentence associated with the multiple speech frames; determine the first predicted quality score corresponding to each speech frame and the second predicted quality score corresponding to each sentence as the predicted quality score of the sample speech signal; A first classification unit, configured to obtain a prediction quality classification of the sample speech signal based on the speech features of each speech frame through the first classification network, wherein the prediction quality classification is used to indicate whether the sample speech signal is a synthetic human voice or a natural human voice; a second classification unit, configured to obtain a predicted source classification of the sample speech signal based on the speech features of each speech frame through the second classification network, wherein the predicted source classification is used to represent a source object of the sample speech signal, and the source object of the sample speech signal is one of M preset source objects, where M is an integer greater than or equal to 2; A parameter adjustment unit is used to calculate the total loss function value of the speech quality evaluation model based on the prediction quality score, the prediction quality classification and the prediction source classification, and adjust the parameters of the speech quality evaluation model according to the total loss function value.

7. A training device for a speech quality evaluation model, characterized in that: It includes a processor, a memory and a communication interface, wherein the processor, the memory and the communication interface are interconnected, wherein the communication interface is used to receive and send data, the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program; when the computer program runs on one or more processors, the method according to any one of claims 1 to 5 is executed.

Images