Conversion device, conversion method, and conversion program
The conversion device improves voice quality by estimating the slope of the target voice's log probability distribution and optimizing for both feature and pattern distances, addressing the limitations of conventional VoiceGrad in preserving phonology and intonation, resulting in clearer and more natural converted speech.
Patent Information
- Application Number
- PCT/JP2023/019403
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-28
AI Technical Summary
Conventional voice quality conversion technologies using VoiceGrad often fail to preserve phonology and intonation, resulting in lower clarity and naturalness of converted speech when optimizing only the log probability distribution of the target voice feature series.
A conversion device and method that estimate the slope of the log probability distribution of the target voice using a trained model from the input voice feature sequence, optimizing an objective function that includes the distance between the input and converted voice feature series to improve clarity and naturalness, employing weighted denoising score matching and annealed Langevin dynamics.
The proposed solution enhances the clarity and naturalness of converted speech by optimizing the objective function with both feature distance and pattern distance, leading to improved voice quality compared to conventional VoiceGrad methods.
Smart Images

Figure JP2023019403_28112024_PF_FP_ABST
Abstract
Description
Conversion device, conversion method, and conversion program
[0001] The disclosed technology relates to a conversion device, a conversion method, and a conversion program.
[0002] Most conventional voice conversion technologies require the preparation of parallel data, which are pairs of speech samples of the source speaker and the target speaker, each uttering the same sentence, as training data. Furthermore, the source speech, or speech data of the source speaker, must be included in the training data.
[0003] In contrast, VoiceGrad, a voice conversion technology based on the principles of score matching and Langevin dynamics, is characterized by its ability to train using non-parallel data and its ability to perform highly accurate voice conversion for any speaker's voice. Specifically, VoiceGrad approaches the conversion process of converting a feature sequence of input speech, which is the source speech, into a feature sequence of target speech as a local optimum solution search process based on the probability distribution of the feature sequence of the target speech. VoiceGrad is based on the idea of training a neural network that estimates the gradient of the logarithmic probability distribution of the feature sequence of the target speech, and iteratively changing the direction of the estimated gradient of the logarithmic probability distribution of the feature sequence of the target speech, using the feature sequence of the input speech as the initial point. VoiceGrad can be implemented using weighted denoising score matching and simulated annealing Langevin dynamics.
[0004] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution", in Advances in Neural Information Processing Systems 32, 2019, pp.11918-11930.
[0005] However, in the conventional VoiceGrad, the objective function to be optimized was only the logarithmic probability distribution obtained by taking the logarithm of the probability density function of the feature sequence of the target speech, so there was no guarantee that the phonemes, intonation, etc. of the input speech would be preserved in the conversion process. Furthermore, in actual VoiceGrad experiments, it was confirmed that the clarity and naturalness of the converted speech, in which the voice quality was converted from the input speech to the target speech, was lower than that of the input speech.
[0006] The disclosed technology has been made in consideration of the above points, and aims to provide a conversion device, a conversion method, and a conversion program that can improve the clarity and naturalness of converted speech compared to when the objective function to be optimized is only the logarithmic probability distribution of the feature sequence of the target speech.
[0007] A first aspect of the present disclosure is a conversion device comprising: an acquisition unit that acquires a gradient of a logarithmic probability distribution of a feature sequence of a target speech estimated from a feature sequence of an input speech before conversion using a trained model; and an output unit that estimates a feature sequence of the converted speech using the gradient of the logarithmic probability distribution of the feature sequence of the target speech so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech and the feature sequence of the converted speech after conversion, and generates and outputs the converted speech.
[0008] A second aspect of the present disclosure is a conversion method, in which an acquisition unit acquires a gradient of a logarithmic probability distribution of a feature sequence of a target speech estimated from a feature sequence of an input speech before conversion using a trained model, and an output unit uses the gradient of the logarithmic probability distribution of the feature sequence of the target speech to estimate a feature sequence of the converted speech so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech and the feature sequence of the converted speech after conversion, and generates and outputs the converted speech.
[0009] A third aspect of the present disclosure is a conversion program for causing a computer to function as each part of the conversion device of the first aspect.
[0010] According to the disclosed technique, the clarity and naturalness of the converted speech can be improved compared to when the objective function to be optimized is only the logarithmic probability distribution of the feature sequence of the target speech.
[0011] FIG. 1 is an example of an algorithm for simulated annealed Langevin dynamics. FIG. 2 is a schematic diagram showing an example of a network structure of a score approximator according to an embodiment. FIG. 3 is a schematic diagram showing an example of a hardware configuration of a conversion device according to an embodiment. FIG. 4 is a block diagram showing an example of a functional configuration of a conversion device according to an embodiment. FIG. 5 is a flowchart showing an example of conversion processing in a conversion device according to an embodiment. FIG. 6 is a schematic diagram showing an example of an evaluation result of an AB test. FIG. 7 is a schematic diagram showing an example of an evaluation result of an ABX test.
[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0013] <Outline of the Embodiment of the Present Invention> <Principle of VoiceGrad> First, the principle of VoiceGrad will be described.
[0014] The gradient ∇ of logp(x), the logarithm of any continuous and differentiable probability density function p(x), with respect to x. x When logp(x) is called the "score function," if the score function of some data x is given, one sample according to the probability density function p(x) can be sampled. Specifically, in the manner of gradient ascent, x (0) is the initial point, and x is calculated as in equation (1) so that logp(x) becomes large. (t) By repeatedly updating p(x), it is possible to obtain one sample according to the probability density function p(x).
[0015] (1)
[0016] In equation (1), α is a positive step size parameter, T is the number of iterations, and z (t) is a Gaussian white noise with a mean of 0 and a variance of 1. The update rule expressed by equation (1) including the noise term expressed by equation (2) is called "Langevin dynamics."
[0017] (2)
[0018] When T is sufficiently large and α is sufficiently small, under certain regularity conditions, x (T) It has been shown that the probability density function p(x) is a sample that follows the probability density function p(x). This idea is applicable to the case where the probability density function p(x) cannot be estimated, but the score function ∇ x This means that if logp(x) can be estimated, it is possible to obtain a sample that follows the probability density function p(x). Therefore, when a training sample X shown in equation (3) is given, the focus of the problem is how to estimate the score function of the training sample X.
[0019] (3)
[0020] Score matching is a method for converting the input voice feature sequence into a target voice quality using a score approximator s θ (x) with respect to the parameter θ. This problem can be solved by, for example, θ (x) and the score function ∇ x This can be formulated as a problem of minimizing the expected value of the squared error between logp(x) and
[0021] (4)
[0022] In equation (4), if the number of training samples X is sufficiently large, equation (5) can be approximated by the sample mean for the training samples X. In the above objective function, the target value ∇ x It is assumed that logp(x) is observable, but several methods have been proposed that can reduce the problem to an equivalent optimization problem without assuming a specific form for the probability density function p(x).
[0023] (5)
[0024] The first method is called implicit score matching, which utilizes the fact that the equation shown in equation (4) is equal to equation (6) except for the constant term.
[0025] (6)
[0026] In addition, in equation (6), ∇ x s θ (x) is the score approximator s θ (x), and tr(·) denotes the trace of the matrix. In this method, the score function ∇ x Instead of being able to eliminate logp(x), we can use tr(∇ x s θ However, the score approximator s θ If (x) is represented by a deep neural network, or if x is high-dimensional, then tr(∇ x s θ Calculating (x) is very expensive.
[0027] In contrast, tr(∇ x s θ Noise-removing score matching (hereinafter referred to as "DSM") can avoid the calculation of (x). DSM is based on the idea of intentionally adding noise to data x according to an appropriate distribution shown in equation (7) and estimating a score function in the distribution of the noisy data shown in equation (8). Note that σ represents a parameter.
[0028] (7)
[0029] (8)
[0030] In equation (8), the distribution q σ (x) can be regarded as a Parzen window estimator of the probability density function p(x). When the noise distribution is a Gaussian distribution shown in equation (9), the objective function to be minimized is shown in equation (10).
[0031] (9)
[0032] (10)
[0033] The score approximator s that minimizes the objective function shown in Equation (10) θ (x) is ∇ x logq σ (x) and the noise variance σ 2 is sufficiently small and the distribution q σ If (x) is approximately equal to the probability density function p(x), then the score approximator s θ (x) is the score function ∇ x It also corresponds to logp(x), i.e., the logarithmic distribution logq σ The direction of the gradient of (x) should coincide with the direction towards x before adding noise at the point shown in equation (11).
[0034] (11)
[0035] Recently, based on the idea of DSM, score approximators (s) θ Attempts have been made to realize high-resolution image generation by representing (x) using a neural network. For example, this attempt is described in detail in Reference 1 below. [Reference 1] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," in Advances in Neural Information Processing Systems 32, 2019, pp. 11918-11930.
[0036] A challenge in applying DSM to image generation tasks is that much of the real-world data, including images, tends to be localized on a low-dimensional manifold in a high-dimensional space. This tendency can be a major problem when simply applying DSM, since DSM assumes that the data distribution is supported over the entire space. In other words, there is no clue to estimate the score at points where there is little data.
[0037] Therefore, a method called weighted DSM was devised as shown in Reference 1. In weighted DSM, a score approximator s is calculated using multiple noise levels shown in Equation (12). θ In the weighted DSM, the initial iteration of the test sampling is q σl The noise level is gradually changed from large to small so that (x) covers the entire space and eventually approaches the true distribution p(x).
[0038] (12)
[0039] Here, the score approximator s θ The score approximator s is conditioned on the noise level l so that (x) can learn different behavior depending on the noise level. θ The key point of this idea is to consider (x, l). As a learning criterion, for example, D σ1 (θ), ..., D σL However, as θ approaches the optimal solution, ||s θ (x, l) || 2 is 1 / σ l Since the weight λ l >0 to σ l 2 It has been proposed to use the value shown in equation (14) set as the learning criterion. It has also been experimentally reported that it is best to use the geometric progression shown in equation (15) for the noise level.
[0040] (13)
[0041] (14)
[0042] (15)
[0043] Under the above settings, the score approximator s θ Once (x, l) is learned, q σL The sample conforming to (x) can be sampled by the iterative algorithm (Simulated Langevin Dynamics) shown in Algorithm 1, where α l is the step size that changes adaptively depending on the noise level, and ε is its scale parameter. Figure 1 shows an example of the algorithm for simulated annealing Langevin dynamics.
[0044] (16)
[0045] In VoiceGrad, the conversion from a feature sequence of an input speech to a feature sequence of a target speech is achieved by iteratively changing the gradient direction of the logarithmic probability distribution of the feature sequence of the target speech, starting from the feature sequence of the input speech as the initial point. One specific implementation method is a technique using weighted DSM and simulated annealed Langevin dynamics. Note that, as will be described in detail later, this method does not guarantee that the speech content of the input speech will be preserved. However, it has been experimentally confirmed that the speech content of the input speech can be preserved depending on the settings.
[0046] In the method using DSM and Langevin dynamics, it is possible to prepare a score approximator for each target speaker. However, in order to predict the scores of feature sequence distributions for K speakers using only a single score approximator, we use a network s conditioned not only by the noise level but also by the speaker index k∈1,...,K}. θ It is also possible to express the score approximator by (x, l, k). The learning criterion in the former case is shown in equation (17), and the learning criterion in the latter case is shown in equation (18).
[0047] (17)
[0048] (18)
[0049] In equation (17), the network parameters have degrees of freedom for each target speaker, and in equation (18), a single network is responsible for predicting the scores of K speakers. k,n When the feature sequence of the n-th utterance of speaker k is expressed by equation (19) where n is the length of the feature sequence, equation (20) can be approximated by the sample average for training data X consisting of N utterances for each speaker, as expressed by equation (21).
[0050] (19)
[0051] (20)
[0052] (21)
[0053] score approximator s θk (x, l) or s θ If (x, l, k) can be learned, the feature sequence of the input voice can be set to the initial point x (0) and the score approximator s θ (x, l) as s θk (x, l) or s θ By replacing the input voice feature sequence with (x, l, k) and then executing Algorithm 1, the voice quality of speaker k can be converted. (t) Since good results were obtained even without adding a noise term in the update of (22), the update rule may be expressed as follows:
[0054] (22)
[0055] The above method does not depend on the probability distribution of the feature sequence of the input speech, and therefore, theoretically, it is applicable to input speech from any speaker.
[0056] <Feature Amount> Any feature amount may be used as long as it is sufficient to ultimately form a speech waveform, and several options are conceivable.
[0057] The first is the vocoder parameters. There are various types of vocoders to choose from, but the Mel-Cepstral Vocoder, which is often used in speech synthesis, is a typical example. The Mel-Cepstral Vocoder uses Mel-Cepstral coefficients for each short interval, a fundamental frequency F 0 , and the aperiodicity index. Therefore, it is possible to use a vector combining these as a feature. However, the fundamental frequency F 0 The fundamental frequency pattern (hereinafter referred to as "F 0 The logarithm of the fundamental frequency (also called the "pattern") is 0 The mean and variance of the logarithm of the target speaker F 0 A simple method can be used, in which the mean and variance of the input speech are shifted and scaled to match each other. Furthermore, since the aperiodicity index of the input speech can be used as is, it is also possible to use only the mel-cepstral coefficients as features. In the experiments described below, a vector with mel-cepstral coefficients as elements is used as a feature.
[0058] Furthermore, many high-quality neural vocoders, including WaveNet, have been proposed in recent years, and it is conceivable to use features designed for use with these. Many of the neural vocoders proposed to date use the mel spectrum for each short section as a feature, so it is conceivable to use the mel spectrum as a feature in line with this.
[0059] The d-th dimension feature in the short-time frame m is expressed as x d,m Then, in the training and testing, the normalized x d,m can also be used.
[0060] (23)
[0061] In addition, in equation (23), ψ d represents the average of the d-th dimension feature in the voiced section, and ζ d represents the standard deviation of the d-th dimension feature in the voiced section. When the normalized feature shown in Equation (23) is used, in the test, appropriate shifting and scaling can be used to adjust the mean and standard deviation of the feature sequence finally generated by Algorithm 1 so that they match those of the target speaker.
[0062] <Network Structure> The network structure of the score approximator can be designed arbitrarily, with no constraints other than the input and output being isomorphic. Figure 2 shows an example of the network structure of a score approximator. In Figure 2, Conv1d represents a one-dimensional convolutional layer, GLU represents a nonlinear activation layer, and Deconv1d represents a one-dimensional transposed convolutional layer. Also in Figure 2, k represents the kernel size of each convolutional layer, c represents the number of output channels, and s represents the stride width. Also in Figure 2, the noise level and speaker index are each represented by a one-hot vector, and a series of these repeated in the time direction is connected in the input channel direction in each convolutional layer. Note that a normalized linear layer, a gated linear layer, or the like can be used as the nonlinear activation layer.
[0063] 3 , a conversion device 10 according to an embodiment of the present invention includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.
[0064] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, a conversion program is stored in the ROM 12 or the storage 14. The conversion program may be a single program, or may be a group of programs composed of multiple programs or modules.
[0065] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.
[0066] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.
[0067] During learning, the input unit 15 receives time-series data of the target voice for each type of target voice as learning data. During conversion processing, the input unit 15 also receives time-series data of the input voice to be converted and information indicating the type of target voice. The type of target voice may be, for example, gender, adult / child, speaker ID, etc.
[0068] The communication I / F 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).
[0069] Next, a description will be given of the functional configuration of the conversion device 10. As shown in Fig. 4, the conversion device 10 includes a learning unit 100, an acquisition unit 101, and an output unit 102.
[0070] The training unit 100 trains a score approximator as a model for estimating the gradient of the logarithmic probability distribution of the feature sequence of the target voice, based on the feature sequence of the target voice, for each type of target voice.
[0071] Specifically, the learning unit 100 extracts a feature sequence of the target voice from the time-series data of the target voice, which is the training data, for each type of target voice, and calculates the gradient of the logarithmic probability distribution of the feature sequence of the target voice from the feature sequence of the target voice. The feature sequence may include, for example, the mel-cepstral coefficient for each short interval, the fundamental frequency F 0 , and a vector of aperiodicity indices.
[0072] Then, for each type of target voice, the learning unit 100 learns a score approximator as a model that estimates the gradient of the logarithmic probability distribution of the feature sequence of the target voice from a combination of the feature sequence of the target voice and the gradient of the logarithmic probability distribution of the feature sequence of the target voice.
[0073] Here, the score approximator, which is a trained model, is a score approximator s trained for each type k of target speech. θk (x, l), or a score approximator s including the type of target speech k as a parameter. θ (x, l, k).
[0074] Then, the learning unit 100 passes the score approximator, which is the trained model, to the acquisition unit 101.
[0075] The acquisition unit 101 acquires the gradient of the logarithmic probability distribution of the feature sequence of the target speech, which is estimated from the feature sequence of the input speech before conversion using a trained model.
[0076] Specifically, the acquisition unit 101 acquires time-series data of the input speech. Then, the acquisition unit 101 extracts a feature sequence from the time-series data of the input speech. Then, based on information indicating the input target speech type k, the acquisition unit 101 estimates the gradient of the logarithmic probability distribution of the feature sequence of the target speech from the feature sequence of the input speech using a score approximator, which is a trained model. At this time, the acquisition unit 101 estimates the gradient of the logarithmic probability distribution of the feature sequence of the target speech from the feature sequence of the input speech using a score approximator s trained in advance for each target speech type k. θk (x, l), or a score approximator s including the type of target speech k as a parameter.θ (x, l, k) is used.
[0077] The acquisition unit 101 then transfers the gradient of the logarithmic probability distribution of the feature sequence of the target voice to the output unit 102 .
[0078] The output unit 102 estimates the feature sequence of the converted speech using the gradient of the logarithmic probability distribution of the feature sequence of the target speech to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech before conversion and the feature sequence of the converted speech after conversion, and generates and outputs the converted speech.
[0079] In this embodiment, the distance between the feature sequence of the input speech and the feature sequence of the converted speech includes a feature distance related to the bottleneck feature output from the speech recognizer or a pattern distance related to the fundamental frequency pattern. In this embodiment, the pattern distance is the distance between a pattern in which the logarithmic fundamental frequency pattern of the input waveform of the input speech is normalized so that the logarithmic fundamental frequency pattern of the target waveform of the target speech is the average and standard deviation of the logarithmic fundamental frequency pattern of the target waveform of the converted speech.
[0080] Specifically, the output unit 102 inputs the feature sequence of the input speech and the feature sequence of the estimated converted speech to a speech recognizer (hereinafter also referred to as "ASR device") represented by a neural network. The output unit 102 then acquires the bottleneck feature of the input speech and the bottleneck feature of the converted speech output from the ASR device. Note that the bottleneck feature (hereinafter also referred to as "ASR (Automatic Speech Recognition) loss") is a feature extracted from a neural network having a bottleneck structure with a reduced number of nodes in the hidden layer.
[0081] Then, the output unit 102 obtains a feature distance based on the bottleneck feature of the input speech and the bottleneck feature of the converted speech.
[0082] The feature distance is expressed by equation (24). In equation (24), x (0)is the feature sequence of the input speech (i.e., the initial point of the iterative updating process), x is the feature sequence of the converted speech (i.e., the update target), and ASR(·) is the bottleneck feature extraction unit of the ASR device. Also, in equation (24), || ... || p denotes the p-norm, where 1≦p<∞ and 0<q<∞.
[0083] (24)
[0084] The output unit 102 outputs a logarithm F 0 The feature sequence of the input speech and the feature sequence of the estimated converted speech are input to the pattern extractor. 0 The pattern extractor acquires the logarithmic fundamental frequency pattern of the normalized input waveform and the logarithmic fundamental frequency pattern of the converted waveform. Here, the normalized logarithmic fundamental frequency pattern of the input waveform is a pattern normalized so that the logarithmic fundamental frequency pattern obtained by taking the logarithm of the fundamental frequency of the input waveform becomes the average and standard deviation of the logarithmic fundamental frequency pattern of the target waveform related to the target voice. The output unit 102 then acquires the distance between the logarithmic fundamental frequency pattern of the normalized input waveform and the logarithmic fundamental frequency pattern of the converted waveform related to the converted voice as the pattern distance. Hereinafter, the pattern distance will be referred to as F 0 Also known as pattern loss.
[0085] The pattern distance is expressed by equation (25). In equation (25), F 0 (・) is the logarithm F 0 Pattern extractor, μ s and σ s is the logarithm F in the voiced section of the input speech 0 The mean and standard deviation of the pattern, μ t and σ t is the logarithm of the voiced section of the target speech. 0 The mean and standard deviation of the patterns are presented.
[0086] (25)
[0087] The output unit 102 also acquires a probability density function p(x) of the feature sequence of the estimated target voice.
[0088] The output unit 102 then estimates a local solution of the feature sequence of the converted speech that optimizes an objective function expressed using the feature distance, pattern distance, and probability density function p(x) of the feature sequence of the target speech. In this embodiment, the objective function used is logp(x)+g(x), which is the sum of the weighted average g(x) of the feature distance and pattern distance and the logarithmic probability distribution obtained by taking the logarithm of the probability density function p(x).
[0089] Specifically, the output unit 102 calculates x by the steepest ascent method as shown in equation (26). (t) By repeatedly updating, a local solution of the feature sequence of the converted speech that optimizes the objective function is estimated. Note that the reason why Equation (26) can be implemented is because the feature distance and the pattern distance are differentiable with respect to x.
[0090] (26)
[0091] In this embodiment, the output unit 102 generates and outputs converted speech using a neural vocoder from the local solution of the feature sequence of the finally estimated converted speech. For example, the output unit 102 outputs the converted speech to a predetermined area of the storage 14 or to a device configured separately from the conversion device 10.
[0092] <Operation of the Conversion Device According to the Embodiment of the Present Invention> Next, the operation of the conversion device 10 according to the embodiment of the present invention will be described.
[0093] 5 is a flowchart showing the flow of the conversion process by the conversion device 10. The conversion process is performed by the CPU 11 reading the conversion program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Here, it is assumed that the learning unit 100 has previously trained the score approximator. It is also assumed that the conversion device 10 receives, via the input unit 15, time-series data of the input speech to be converted and information indicating the type k of the target speech.
[0094] In step S100, the CPU 11, functioning as the acquisition unit 101, acquires the gradient of the logarithmic probability distribution of the feature sequence of the target speech, estimated from the feature sequence of the input speech using a trained model. Specifically, the CPU 11 extracts a feature sequence from the time-series data of the input speech, and estimates the gradient of the logarithmic probability distribution of the feature sequence of the target speech from the feature sequence of the input speech, based on information indicating the type k of the input target speech, using a score approximator, which is a trained model.
[0095] In step S101, the CPU 11 acquires a feature distance as the output unit 102. Specifically, the CPU 11 inputs the feature sequence of the input speech and the feature sequence of the converted speech estimated in step S104 (described later) to the ASR device, and acquires a feature distance based on the bottleneck feature of the input speech and the bottleneck feature of the converted speech output from the ASR device.
[0096] In step S102, the CPU 11 acquires the pattern distance as the output unit 102. Specifically, the CPU 11 acquires the pattern distance as a logarithm F 0 The feature sequence of the input speech and the feature sequence of the converted speech estimated in step S104 (to be described later) are input to the pattern extractor, and a logarithm F 0 The distance between the pattern of the logarithmic fundamental frequency of the normalized input waveform and the pattern of the logarithmic fundamental frequency of the converted waveform, which is output from the pattern extractor, is obtained as the pattern distance.
[0097] In step S103, the CPU 11 acquires the probability density function p(x) of the feature sequence of the estimated target voice as the output unit 102. Note that the order of the processes from step S101 to step S103 is not limited to the order described above.
[0098] In step S104, the CPU 11, as the output unit 102, estimates a local solution of the feature sequence of the converted speech that optimizes an objective function expressed using the feature distance, pattern distance, and probability density function p(x) in the feature sequence of the target speech. Specifically, the CPU 11 estimates a local solution of the feature sequence of the converted speech that optimizes logp(x)+g(x), which is the sum of the weighted average g(x) of the feature distance shown in equation (24) above and the pattern distance shown in equation (25) above and logp(x), which is the logarithm of the probability density function p(x).
[0099] In step S105, the CPU 11, functioning as the output unit 102, determines whether or not to end the iteration. For example, if the number of iterations has not reached the upper limit, it determines not to end the iteration, and the process returns to step S101. On the other hand, if the number of iterations has reached the upper limit, it determines to end the iteration, and the process proceeds to step S106.
[0100] In step S106, the CPU 11 causes the output unit 102 to generate and output converted speech from the local solutions of the feature sequence of the finally estimated converted speech using a neural vocoder.
[0101] <Experimental Results> <Experimental Conditions> To confirm the effectiveness of voice conversion using the method of the embodiment of the present invention, a voice conversion experiment was conducted using speech data from four speakers (female speaker clb, male speaker bdl, female speaker slt, male speaker rms) from the CMU ARCTIC database for training and testing assuming known speakers. As described above, the number of speakers used for training was four, so the dimension of the one-hot vector representing the speaker index was four. The database consisted of speech samples of the same 1,132 sentences spoken by each speaker, but these sentences were divided into a first half of 1,000 sentences and a second half of 132 sentences, and the speech samples of each speaker from the second half of 132 sentences were used as test data. The sampling frequency of all speech signals was 16,000 Hz.
[0102] In the baseline method and the inventive method, the feature to be converted was the logarithmic mel spectrum. Specifically, for each utterance, 80 channels of logarithmic mel spectrum were extracted at 10 ms intervals. Therefore, the dimension D of the feature is 80.
[0103] In addition, the fundamental frequency F 0 Then, the logarithm F 0 Average m s and standard deviation σ s Calculate the logarithm F in the voiced section from the training data of the target speech. 0 Average m t and standard deviation σ t The network configuration of the score approximator was as shown in Figure 2.
[0104] Regarding hyperparameters, Adam was used to train the score approximator, and the learning rate was set to 0.0005. 1≦l≦L is σ 2 / σ 1 =...=σ L / σ L-1 is approximated to 0.81, L is set to 21, and σ 1 is set to 1.2, and σ L In Algorithm 1, ε is set to 10 -5 and T was set to 32.
[0105] <Baseline Method> For comparison, VoiceGrad was used as a baseline, which had the same conditions except that feature distance and pattern distance were not included during testing.
[0106] <Objective Evaluation Index> Because the test set consists of speech samples in which each speaker speaks the same sentence, the quality of the converted speech can be evaluated by comparing it with the target speech, which is the speech of the target speaker who spoke the same sentence. Mel-Cepstral Distortion (MCD) shown in equation (27) can be used as a measure of the degree of difference between the converted speech and the target speech.
[0107] (27)
[0108] Since the phonemes of the converted speech and the target speech do not necessarily correspond at the same time, the average MCD was calculated after aligning the time axis for each utterance using dynamic time warping (DTW) based on the MCD criterion. 0 As an evaluation index for the pattern, the logarithm F of the converted speech and the target speech is calculated based on the time correspondence curve obtained by the DTW of the MCD standard. 0 The correlation coefficient between the two patterns was calculated after time-warping. 0 Pattern correlation coefficient LFC (Log-F 0 This is called the Contour Correlation Coefficient.
[0109] <Results of Objective Evaluation Experiment> In the invention method, "+ASR" is MCD and LFC that include only ASR loss (feature distance), and F 0 "+F" is the MCD and LFC that include only the pattern loss (pattern distance). 0 " and ASR loss and F 0 MCD and LFC including pattern loss "+ASR+F 0 A comparison between "VoicGrad" which does not include either MCD or LFC is shown in Tables 1 and 2. As shown in Tables 1 and 2, the ASR loss and F 0 By including pattern loss, improvements were observed in all indicators, demonstrating the effectiveness of the present invention.
[0110]
[0111]
[0112] <Results of subjective evaluation experiment> In addition to the objective evaluation experiment, subjective evaluation experiments were also conducted using the AB test for sound quality evaluation and the ABX test for speaker similarity evaluation. In the sound quality evaluation, "VoiceGrad" and VoiceGrad plus ASR loss and F 0 Including pattern loss "+ASR+F 0The quality of the sound quality of pairs of samples randomly selected from the converted speech generated using " and " was evaluated as A (Sure), A (Not sure), B (Not sure), or B (Sure). The evaluation results are shown in Figure 6.
[0113] In addition, in the speaker similarity evaluation, "VoiceGrad" and "+ASR+F" are also used. 0 Participants listened to a pair of randomly selected samples from the converted speech generated using the ' and ' samples, along with the target speaker's actual speech sample, and evaluated whether the two speeches were likely to have been produced by the same speaker using A (Sure), A (Not sure), B (Not sure), or B (Sure). The evaluation results are shown in Figure 7.
[0114] As shown in FIGS. 6 and 7, the "+ASR+F" method according to the embodiment of the present invention 0 " received higher ratings in sound quality and speaker similarity than the baseline method "VoiceGrad."
[0115] <Summary> As described above, in the conversion device 10 according to this embodiment, the acquisition unit 101 acquires the gradient of the logarithmic probability distribution of the feature sequence of the target speech, estimated from the feature sequence of the input speech before conversion using a trained model. The output unit 102 estimates the feature sequence of the converted speech using the gradient of the logarithmic probability distribution of the feature sequence of the target speech to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech and the feature sequence of the converted speech after conversion, and generates and outputs the converted speech. This improves the clarity and naturalness of the converted speech compared to when the objective function to be optimized is only the logarithmic probability distribution of the feature sequence of the target speech.
[0116] In the process of converting a feature sequence of an input speech into a feature sequence of a target speech, one possible method for preserving the phonemes, intonation, etc. of the input speech is to constrain the ASR results and the relative fundamental frequency pattern obtained by normalizing the mean and variance of the logarithmic fundamental frequency pattern in the voiced section so that the input speech and the converted speech match. VoiceGrad allows for flexible design of update rules for obtaining speech that simultaneously satisfies one or more requirements other than increasing the logarithmic probability distribution logp(x), provided that these requirements can be defined as differentiable functions. Therefore, in this embodiment, as described above, the conversion device 10 uses Equation (26) to output a local solution of the converted speech that optimizes the objective function using the steepest ascent method.
[0117] <Modifications> The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.
[0118] For example, in the above-described embodiment, the learning unit 100 of the conversion device 10 learns the score approximator. However, this is not limiting. For example, the score approximator may be learned by a device other than the conversion device 10.
[0119] Furthermore, the various processes executed by the CPU after reading the software (program) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processes. Furthermore, the conversion process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.
[0120] In the above embodiment, the conversion program is pre-stored (installed) in the ROM 12 or the storage 14, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.
[0121] The following additional notes are provided regarding the above-described embodiments.
[0122] (Supplementary Item 1) A conversion device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: obtain a gradient of a logarithmic probability distribution of a feature sequence of a target speech estimated from a feature sequence of an input speech before conversion using a trained model; estimate a feature sequence of the converted speech using the gradient of the logarithmic probability distribution of the feature sequence of the target speech so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech and the feature sequence of the converted speech after conversion; and generate and output the converted speech.
[0123] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform a conversion process, wherein the conversion process: obtains a gradient of a logarithmic probability distribution of a feature sequence of a target speech estimated using a trained model from a feature sequence of an input speech before conversion; and an output unit uses the gradient of the logarithmic probability distribution of the feature sequence of the target speech to estimate a feature sequence of the converted speech so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target speech and the distance between the feature sequence of the input speech and the feature sequence of the converted speech after conversion; and generates and outputs the converted speech.
[0124] REFERENCE SIGNS LIST 10 Conversion device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 17 Communication I / F 19 Bus 100 Learning unit 101 Acquisition unit 102 Output unit
Claims
1. A conversion device comprising: an acquisition unit that acquires a gradient of a logarithmic probability distribution of a feature sequence of a target voice estimated from a feature sequence of an input voice before conversion using a trained model; and an output unit that estimates a feature sequence of the converted voice using the gradient of the logarithmic probability distribution of the feature sequence of the target voice so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target voice and the distance between the feature sequence of the input voice and the feature sequence of the converted voice after conversion, and generates and outputs the converted voice.
2. The conversion device according to claim 1, further comprising a learning unit configured to learn a model for estimating the gradient of the logarithmic probability distribution of the feature sequence of the target voice for each type of the target voice, based on the feature sequence of the target voice.
3. A conversion device as described in claim 1, wherein the distance between the feature sequence of the input speech and the feature sequence of the converted speech includes a feature distance related to a bottleneck feature output from a speech recognizer, or a pattern distance related to a fundamental frequency pattern.
4. The conversion device according to claim 1, wherein the trained model is a model trained in advance for each type of target voice, or a model including the type of target voice as a parameter.
5. The conversion device according to claim 3, wherein the objective function includes a sum of a weighted average of the feature distance and the pattern distance and a logarithmic probability distribution of the feature sequence of the target voice.
6. The conversion device described in claim 3, wherein the pattern distance is the distance between a pattern of the logarithmic fundamental frequency of the input waveform related to the input voice, normalized so that the pattern of the logarithmic fundamental frequency is the average and standard deviation of the pattern of the logarithmic fundamental frequency of the target waveform related to the target voice, and the pattern of the logarithmic fundamental frequency of the converted waveform related to the converted voice.
7. A conversion method, in which an acquisition unit acquires a gradient of a logarithmic probability distribution of a feature sequence of a target voice estimated from the feature sequence of an input voice before conversion using a trained model, and an output unit estimates a feature sequence of the converted voice using the gradient of the logarithmic probability distribution of the feature sequence of the target voice so as to optimize an objective function expressed using the logarithmic probability distribution of the feature sequence of the target voice and the distance between the feature sequence of the input voice and the feature sequence of the converted voice after conversion, and generates and outputs the converted voice.
8. A conversion program for causing a computer to function as each part of the conversion device according to any one of claims 1 to 6.
Citation Information
Patent Citations
Vocal quality conversion method and vocal quality conversion device
JP2017003622A
Voice conversion model learning device, voice conversion device, method, and program
JP2018136430A
Learning method of conversion model and learning device of conversion model
JP2019040123A
High-quality non-parallel many-to-many voice conversion
US20200365166A1
Voice quality conversion device, voice quality conversion method and program
WO2018159612A1
