Voice subjective evaluation value estimation device and voice subjective evaluation value estimation method
Patent Information
- Application Number
- JP2025538918
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Filing Date
- 2023-08-04
- Publication Date
- 2025-02-13
Abstract
Description
Apparatus and method for estimating subjective speech assessment scores
[0001] The present invention relates to a speech subjective assessment value estimation device and a speech subjective assessment value estimation method.
[0002] Text-to-speech synthesis (TTS) is a technology that converts text into speech waveforms. Voice conversion (VC) is a system that converts non-verbal information from speech into another person's voice while preserving linguistic information. Many highly accurate TTS and VC methods based on deep learning have been proposed. Some TTS and VC methods not only reproduce the voice of a specific person, but also control the speaking style, such as TTS that reproduces specific emotions, TTS that can control the output voice, and VC that reproduces specific emotions.
[0003] Deep learning-based SQA (Speech Quality Assessment) methods for predicting subjective ratings of speech quality after TTS and after VC are also being developed (Non-Patent Document 1). SQA methods use subjective rating data as training data and are being researched with the aim of avoiding the need for large-scale subjective rating surveys for performance evaluation of audio equipment and speech processing systems. The absolute category scaling (ACR) method specified in ITU-T Recommendation P.910 is primarily used as the format of options presented in the subjective rating questionnaire used to collect training data for SQA methods. ACR uses five options: "1: bad," "2: poor," "3: fair," "4: good," and "5: excellent." The MOSNet disclosed in Non-Patent Document 1 uses mean opinion scores (MOS), which are the average of the ACR-style ratings assigned by subjects to specific speech data, as training data, and the MOSNet network is trained to predict MOS values.
[0004] The above technology predicts subjective voice quality, and the voices people desire may differ depending on the situation. Various evaluation criteria for evaluating voices can be considered to meet the needs of each situation. For example, evaluation criteria such as "cute" or "clear" could be considered.
[0005] C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, and HM Wang. MOSNet:Deep learning based objective assessment for voice conversion. In Proc. Interspeech, 2019. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proc. ICML, 2005.
[0006] However, when using evaluation criteria such as "cute" or "full," it is expected that evaluators will not agree on the merits of two sounds. For such everyday expressions, it is difficult to apply some of the survey design techniques specified in ITU-T and ITU-R recommendations to reduce inaccuracies and biases in labeling results.
[0007] In view of the above circumstances, an object of the present invention is to provide a technique capable of estimating a subjective evaluation value of speech data based on everyday expressions.
[0008] One aspect of the present invention is a speech subjective assessment estimation device that inputs speech data into an estimation model and outputs a subjective assessment value of the speech data, wherein the estimation model is a model trained on the basis of a data set in which speech data and subjective assessment values of the speech data are associated with each other.
[0009] According to the present invention, it is possible to estimate a subjective evaluation value of speech data based on everyday expressions.
[0010] 1 is a diagram illustrating an example of the configuration of an estimation model generation device; FIG. 2 is a diagram illustrating an example of the architecture of an estimation model; FIG. 3 is a diagram illustrating an example of another training method for an estimation model; FIG. 4 is a diagram illustrating an example of the configuration of a speech subjective assessment estimation device; FIG. 5 is a diagram illustrating the number of speakers in the performance evaluation data for each judgment criterion, the number of speakers in the training data for the pointwise approach, and the number of speakers in the training data for the pairwise approach; FIG. 6 is a diagram illustrating the maximum average ppref of all ppref trials for each approach and the standard deviation of ppref in the epoch at which the average ppref is maximum; FIG. 7 is a diagram illustrating the average ppref of each epoch for "young-sounding voice (male)"; FIG. 8 is a diagram illustrating the average value and standard deviation of the maximum ppref of each trial; FIG. 9 is a diagram illustrating the average number of epochs at which ppref is maximized in each trial; and FIG. 10 is a diagram illustrating the average training time per epoch for each approach.
[0011] 1 is a diagram showing an example of the configuration of an estimation model generating device 10. The estimation model generating device 10 includes a teacher data acquiring unit 101, a learning unit 102, and a model output unit 103.
[0012] The training data acquisition unit 101 acquires training data. The training data includes speech data x and labels attached to the speech data x. The speech data x is, for example, a mel spectrogram.
[0013] The learning unit 102 learns an estimation model, which receives speech data x and outputs a subjective evaluation score m of the speech data.
[0014] Figure 2 is a diagram showing an example of the architecture of an estimation model. The estimation model is configured, for example, by combining a convolutional neural network (CNN) and a bidirectional long short-term memory (BLSTM). In the estimation model shown in Figure 2, speech data x is input to a total of 12 layers of CNN (shown as Conv2d in Figure 2) and a ReLU layer, and then input to a BLSTM layer, a fully connected layer (FC), a ReLU layer, a Dropout layer, and an FC layer, and finally average pooling is applied to output a score m.
[0015] The training data acquisition unit 101 acquires training data in which speech data x is associated with a score m indicating a subjective evaluation based on a predetermined evaluation criterion corresponding to the speech data x. The learning unit 102 trains the estimation model based on a score output by inputting the speech data x to the estimation model and a score corresponding to the speech data x in the training data. ｉ The score m is output by inputting ｉ and audio data x ｉ The score m corresponding to ＾ ｉ From equation (1), the mean square error (MSE) L ＭＳＥ The parameters of the estimation model are updated by calculating and performing error backpropagation.
[0016] The above estimation model learning method will be referred to as the pointwise approach hereinafter.
[0017] 3 is a diagram showing an example of another learning method for an estimation model. The learning method shown in FIG. 3 is the RankNet method disclosed in Non-Patent Document 2. In the learning method shown in FIG. 3, a training data acquisition unit 101 acquires two pieces of speech data x ｉ and x ｊ and pairwise comparison label c between two audio data ｉ，ｊ The training data associated with the paired comparison label c is obtained. ｉ，ｊ is audio data x ｉ and x ｊ The learning unit 102 compares two pieces of speech data xｉ and x ｊ The difference in score calculated from ｉ，ｊ The voice data x calculated based on ｉ More audio data x ｊ is superior to P ｉ，ｊ and pairwise comparison label c ｉ，ｊ The voice data x calculated based on ｉ More audio data x ｊ is superior to P ＾ ｉ，ｊ The score of the speech data x is calculated by the estimation model having the architecture shown in FIG.
[0018] For example, the learning unit 102 calculates the score Δ ｉ，ｊ Calculate f θ (x) is the output when speech data x is input to the estimation model.
[0019] The learning unit 102 calculates the score Δ ｉ，ｊ is input to the sigmoid function shown in equation (3), the probability P ｉ，ｊ Calculate.
[0020] Audio data x ｉ More audio data x ｊ is superior to P ＾ ｉ，ｊ is the pairwise comparison label c ｉ，ｊ The way of making this correspondence is arbitrary, but generally the labels are determined according to the audio data x ｊ If the value indicates that the voice data x is superior, it corresponds to a value close to 1. i A value close to 0 indicates superiority.
[0021] The learning unit 102 ｉ，ｊ and P ＾ ｉ，ｊ is expressed as the binary cross entropy L ＢＣＥ The parameters of the estimation model are updated by performing backpropagation using the loss function.
[0022] The above learning method will be referred to hereinafter as the pairwise approach.
[0023] The model output unit 103 outputs the learned estimation model.
[0024] 4 is a diagram showing an example of the configuration of the speech subjective assessment value estimation device 11. The speech subjective assessment value estimation device 11 includes a data acquisition unit 111, a score calculation unit 112, a result output unit 113, a judgment result output unit 114, and a storage unit 119. The storage unit 119 stores the estimation model output by the estimation model generation device 10.
[0025] The data acquisition unit 111 acquires a feature quantity x of the audio data. The feature quantity x of the audio data is, for example, a mel spectrogram. The data acquisition unit 111 may acquire raw data of the audio data and convert it into a mel spectrogram. The data acquisition unit 111 acquires the feature quantities x of the two audio data. ｉ , x ｊ may be obtained.
[0026] The score calculation unit 112 calculates the score m by inputting the speech data x to the estimation model. ｉ , x ｊ When acquiring the score, the score calculation unit 112 calculates the score using two pieces of audio data x ｉ , x ｊ may be input to the estimation model, two scores may be calculated, and the audio data with the larger score may be determined.
[0027] The result output unit 113 outputs the score m. ｉ , x ｊ When obtaining the score, the result output unit 113 may output the voice data with the highest score.
[0028] (Experimental Results) The experiments conducted are described below. The feature value of the speech data used was a 257-dimensional mel spectrogram. The original speech data was stored in the ATR Multi-Speaker Speech Database, and consisted of Japanese speech of 15 different sentences read by 786 male speakers and 1,351 female speakers. The average length of the speech was 5 seconds. Each speech was normalized with respect to the average sound pressure excluding silent intervals. The sampling frequency of the mel spectrogram was 16 kHz, and a Hamming window was used as the window function, with the window length and shift length being 32 ms and 16 ms, respectively.
[0029] Fifty subjects (25 men and 25 women) were asked to assign subjective evaluation labels to the audio. The subjective evaluation labels used to train the model differ between the pointwise and pairwise approaches. In the pointwise approach, subjects were asked to play one audio clip and assign a five-point score m based on a predetermined evaluation criterion. The five-point score m was assigned in descending order of excellence: 5 (excellent), 4 (slightly excellent), 3 (average), 2 (not very excellent), and 1 (poor).
[0030] In the pairwise approach, two speeches i and j are played to the subject in turn, and a pairwise comparison label c is generated based on a predetermined evaluation criterion. ｉ，ｊ The four-level paired comparison label c ｉ，ｊ means "i is better," "i is slightly better," "j is slightly better," or "j is better."
[0031] The number of evaluations for the pointwise approach was approximately 27,000, and the number of evaluations for the pairwise approach was approximately 13,000. In the pairwise approach, the two audio streams played were the same sentences, and were voiced by the same gender. The evaluation criteria used for female voices were "youthful-sounding female voice," and for male voices were "youthful-sounding male voice" and "resonant male voice."
[0032] The estimation model used has the architecture shown in Fig. 2. The learning method of the estimation model in the point-wise approach and pair-wise approach is the same as that in the example of the above embodiment. That is, in the point-wise approach, the mean square error (MSE) L ＭＳＥ The parameters of the estimation model were updated by backpropagating the error. In the pairwise approach, the binary cross entropy L ＢＣＥ The parameters of the estimation model were updated by backpropagating the error using the loss function c. ｉ，ｊ Probability P based on ＾ ｉ，ｊ In this example, the values were 1, w, 1-w, and 0, in the order of "i is better," "i is slightly better," "j is slightly better," and "j is better." Here, the hyperparameter w is a preset value in the range of 0<w<0.5. In this experiment, the values of 0.2 and 0.3 were used for the hyperparameter w.
[0033] The parameters of the estimation model were initialized using a standard normal distribution. Adam was used for optimization, with a learning rate of 0.0001. The batch size was 32. The number of epochs for learning was 100. The order in which the parameters of the estimation model and the training data were learned was randomly changed, and 27 trials were performed.
[0034] 652 pieces of evaluation data were randomly selected from data labeled in the form of paired comparisons and used as evaluation data for evaluating the estimation accuracy of the speech subjective assessment value estimation device 11. For the evaluation data, "i is better" and "i is slightly better" were treated as "i is better," and "j is slightly better" and "j is better" were treated as "j is better," thereby indicating the superiority or inferiority of the two speech data.
[0035] The training data was randomly selected to ensure there was no overlap between the speech data included in the training data and the speech data included in the evaluation data. The number of speech data items included in the training data for "young-sounding voice (female)" was 7,962 for both approaches, and 7,690 for "young-sounding voice (male)" and "strong voice" for both approaches. Figure 5 shows the number of speakers in the performance evaluation data for each criterion, the number of speakers in the training data for the pointwise approach, and the number of speakers in the training data for the pairwise approach.
[0036] To compare the estimation accuracy, we used ppref (precision of preferences), which is defined by equation (5).
[0037] N all is the number of pairs of speech data and pairwise comparison labels included in the evaluation data, and N correct is the number of pairs of speech data and paired comparison labels included in the evaluation data, in which the superior speech data indicated by the paired comparison label corresponding to the speech data is the same as the superior speech data estimated by the speech subjective assessment value estimation device 11 based on the speech data. The estimation result by the speech subjective assessment value estimation device 11 is the speech data with the higher score m in the point-wise approach, and is the speech data indicated as superior by the pair-wise comparison label in the pair-wise approach.
[0038] Figure 6 shows the maximum average ppref across all trials for each approach, as well as the standard deviation of ppref at the epoch when the average ppref was maximum. Based on experimental results showing that the loudness of F0 in female voices contributes to the impression of youth, we used F0 comparison-based speech preference prediction as the baseline method for "young-sounding voices (female)." F0 was estimated using pYIN.
[0039] The pairwise and pointwise approaches were shown to have comparable estimation performance for "young-sounding voice (female)" and "strong voice." Although pairwise comparison data is easier to label than ACR-format data, it is not inferior as training data. Furthermore, for "young-sounding voice (male)," where the maximum average ppref of the pointwise approach is lower than the other evaluation criteria, the pairwise approach achieved higher estimation accuracy than the pointwise approach. It was also confirmed that the maximum average ppref of the pairwise approach did not vary significantly depending on the value of w.
[0040] Figure 7 shows the average ppref for each epoch for "young-sounding voice (male)." For comparison, we also show the ppref for the pointwise approach, which uses twice the number of training data (15,380). We can see that the ppref for the pairwise approach is equal to or better than that for the pointwise approach for most epochs.
[0041] Figure 8 shows the average and standard deviation of the maximum ppref for each trial. For the "young-sounding voice (male)" category, the maximum ppref in the pairwise approach exceeded that in the pointwise approach, and was also equivalent to that in the pointwise approach for the other evaluation criteria.
[0042] Figure 9 shows the average number of epochs required to maximize ppref in each trial. Figure 10 shows the average training time per epoch for each approach. The pairwise approach requires approximately twice the training time per epoch of the pointwise approach, but the average number of epochs required to reach maximum ppref is approximately half. Therefore, it can be said that there is not much difference between the pairwise and pointwise approaches in the time required to reach maximum ppref.
[0043] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.
[0044] The estimation model generating device 10 and the speech subjective assessment value estimating device 11 in the above-described embodiments may be partially or entirely implemented by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes an operating system (OS) and peripheral hardware. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system. Furthermore, part or all of the estimation model generating device 10 and the speech subjective assessment value estimating device 11 may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).
[0045] 10 Estimation model generation device, 101 Teacher data acquisition unit, 102 Learning unit, 103 Model output unit, 11 Speech subjective assessment value estimation device, 111 Data acquisition unit, 112 Score calculation unit, 113 Result output unit, 119 Storage unit
Claims
1. A speech subjective assessment value estimation device that estimates a subjective assessment value of speech data by inputting the speech data into an estimation model, the estimation model being a model trained on the basis of a data set in which speech data and subjective assessments of the speech data are associated with each other.
2. A speech subjective assessment value estimation device as described in claim 1, which outputs speech data with a high subjective assessment by inputting two pieces of speech data into an estimation model, wherein the estimation model is a model trained on the basis of a data set in which the two pieces of speech data are associated with data indicating the speech data with a higher subjective assessment out of the two pieces of speech data.
3. The speech subjective assessment value estimation device according to claim 1, wherein the estimation model is created by combining a convolutional neural network (CNN) and a bidirectional long short-term memory (BLSTM).
4. A method for estimating a subjective assessment value of speech, comprising inputting speech data into an estimation model to estimate a subjective assessment value of the speech data, the estimation model being a model trained on the basis of a data set in which speech data and subjective assessments of the speech data are associated with each other.