Learning device, conversion device, learning method, and program

Abstract

A learning device comprises a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that performs signal conversion using a diffusion model and is trained through non-parallel conversion. A student model in the knowledge distillation is a mathematical model that performs signal conversion using a diffusion model and has fewer steps in a de-spreading process than the teacher model. In the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal conversion performed by the student model and the result of signal conversion performed by the teacher model.

Description

Learning device, conversion device, learning method and program

[0001] The present invention relates to a learning device, a conversion device, a learning method, and a program.

[0002] Speech conversion is a technology that converts only non-linguistic and paralinguistic information from input speech while preserving the linguistic information. Speech conversion can be applied to a variety of technologies, including speaker characteristics, emotion, and pronunciation conversion, as well as speech enhancement and speech assistance. Training methods for speech conversion can be broadly divided into parallel speech conversion and non-parallel speech conversion.

[0003] Parallel speech conversion is a training method that uses paired data of input and output speech as training data. Parallel speech conversion has the advantage of being easy to train because it can explicitly obtain the correct target speech for the converted speech. However, training requires collecting paired data of input and output speech with the same speech content, which has the disadvantage of high training data collection costs.

[0004] Non-parallel speech conversion has been proposed as a method to overcome the above problems. Non-parallel speech conversion is a speech conversion technology that allows training without using paired data of input speech and output speech. This eliminates the need to collect paired data of input speech and output speech with the same speech content when training, making it possible to reduce the cost of collecting training data.

[0005] On the other hand, unlike parallel speech conversion, it is not possible to explicitly obtain the correct target speech for the converted speech, so training is not easy.

[0006] As a method for solving this problem, non-parallel speech conversion using a diffusion model has been proposed (see Non-Patent Document 1). In the diffusion model, the process of converting data into latent variables (noise) (diffusion process) is realized by gradually adding noise, while the process of converting the latent variables (noise) into data (de-diffusion process) is realized by gradually removing noise.

[0007] Since training is performed to maximize the logarithmic likelihood when diffusion and de-diffusion are performed on each piece of speech data that is not paired, training is possible even if there is no paired data of input speech and output speech with the same speech content.In addition, by performing repeated calculations, the diffusion model can express complex functions that are difficult to express in a single calculation, making it possible to perform high-quality speech conversion.

[0008] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo, and Shogo Seki. VoiceGrad: Non-parallel any-to-many voice conversion with annealed Langevin dynamics. IEEE / ACM Trans. Audio Speech Lang. Process., vol. 32, pp. 2213-2226, 2024.

[0009] However, the calculation time increases in proportion to the number of iterative calculations, resulting in slow conversion speed.

[0010] In view of the above circumstances, an object of the present invention is to provide a technique for increasing the conversion speed.

[0011] One aspect of the present invention is a learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms a signal using a diffusion model and that has been learned through non-parallel transformation; a student model in the knowledge distillation is a mathematical model that transforms a signal using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and the student model is updated in the knowledge distillation so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

[0012] One aspect of the present invention is a conversion device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms a signal using a diffusion model and that has been learned by non-parallel transformation; wherein the student model in the knowledge distillation is a mathematical model that transforms a signal using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and wherein, in the knowledge distillation, the student model is updated so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model; and a conversion unit that performs conversion using a knowledge-distilled student model obtained by a learning device.

[0013] One aspect of the present invention is a learning method executed by a learning device, the learning method including a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model, and the student model is updated in the knowledge distillation so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model, the learning method including a knowledge distillation step in which the control unit performs the knowledge distillation.

[0014] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0015] The present invention makes it possible to increase the conversion speed.

[0016] 1 is an explanatory diagram illustrating a conversion system according to an embodiment; a flowchart illustrating an example of a processing flow executed by a control unit according to an embodiment; a diagram illustrating algorithm 1 according to an embodiment; a diagram illustrating an example of a hardware configuration of a learning device according to an embodiment; a diagram illustrating an example of a hardware configuration of a conversion device according to an embodiment; a flowchart illustrating an example of a processing flow executed by a conversion device according to an embodiment; and a diagram illustrating an example of an experimental result according to an embodiment.

[0017] 1 is an explanatory diagram illustrating a conversion system 100 according to an embodiment. The conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 includes a control unit 11 including a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus. The control unit 11 executes knowledge distillation processing.

[0018] The knowledge distillation process is a process of performing knowledge distillation using a mathematical model that performs speech conversion using a diffusion model and that has been trained by non-parallel speech conversion as a teacher model. Therefore, the teacher model used in this knowledge distillation is a mathematical model that performs speech conversion using a diffusion model and that has been trained by non-parallel speech conversion. Furthermore, the student model in this knowledge distillation is a mathematical model that performs speech conversion using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model.

[0019] The conversion device 2 performs a conversion process, which is a process of converting speech using the student model obtained by the learning device 1 after knowledge distillation.

[0020] The term "knowledge distillation completed" means that knowledge distillation has been performed until a predetermined condition for terminating knowledge distillation (hereinafter referred to as "knowledge distillation termination condition") is satisfied.

[0021] <Effects of knowledge distillation processing> A student model has fewer steps in the dediffusion process than a teacher model. Therefore, a knowledge-distilled student model can perform speech conversion with fewer steps than a teacher model, with almost the same accuracy as that of conversion using a teacher model. Therefore, a knowledge-distilled student model can increase the conversion speed.

[0022] 2 is a flowchart showing an example of the flow of processing executed by the control unit 11 in an embodiment. More specifically, it is a flowchart showing an example of the flow of processing executed in the knowledge distillation process in an embodiment. The control unit 11 acquires a conversion target that is a conversion target by a student model (hereinafter referred to as a "student conversion target") from the training data (step S101). Specifically, the student conversion target is one or more voice signals (voice data) or one or more feature quantities of a voice signal. Next, the control unit 11 performs voice conversion on the student conversion target using the student model (step S102).

[0023] Next, the control unit 11 performs voice conversion on the result of the voice conversion performed in step S102 (hereinafter referred to as the "student result") using the teacher model (step S103). Hereinafter, the result of voice conversion by the teacher model will be referred to as the teacher result. Note that since the teacher model is a diffusion model, in step S103 the teacher model first performs a diffusion process on the student result, and then performs a de-diffusion process on the result of the diffusion process. The result of this de-diffusion process is the teacher result.

[0024] The control unit 11 updates the student model in accordance with a predetermined rule (step S104).

[0025] After step S104, the control unit 11 determines whether a knowledge distillation termination condition is satisfied (step S105). The knowledge distillation termination condition may be any condition related to the termination of knowledge distillation, such as a condition that the student model has been updated a predetermined number of times, a condition that the change in the student model due to the update is smaller than a predetermined change, or a condition that the score satisfies a predetermined condition related to the value of a predetermined index for evaluating the performance of the student model. An example of the condition that the score satisfies a predetermined condition related to the value of a predetermined index for evaluating the performance of the student model is a condition that the difference between the student result and the teacher result is smaller than a predetermined difference.

[0026] If the knowledge distillation termination condition is satisfied (step S105: YES), the knowledge distillation process ends. On the other hand, if the knowledge distillation termination condition is not satisfied (step S105: NO), the process returns to step S101 after step S105.

[0027] <Regarding Loss Function> In the above knowledge distillation, any loss function may be used as long as knowledge distillation from a teacher model to a student model is performed. For example, the loss function may include a loss function based on a diffusion model used by the student model (hereinafter referred to as "diffusion loss"). Specifically, the diffusion loss is a value indicating the difference between the teacher result and the diffusion result. Furthermore, the loss function used in the above knowledge distillation may include a loss function based on adversarial learning (hereinafter referred to as "adversarial loss") in addition to the diffusion loss.

[0028] <Regarding Initial Values ​​of Student Model> Any value may be used as the initial value of the parameter of the student model in the above knowledge distillation as long as knowledge distillation is performed from the teacher model to the student model. For example, the value of the parameter of the teacher model may be used.

[0029] <Regarding Audio Signal Features> The audio signal features described above will now be explained. The audio signal features may be quantities extracted using a predetermined function. The predetermined function may be, for example, a transformation from a waveform to a spectrogram based on basis functions such as a short-time Fourier transform or a wavelet transform. The predetermined function may be, for example, a transformation into a cepstrum obtained by further performing Fourier analysis on the spectrogram, or a transformation into acoustic features such as a fundamental frequency, an asynchrony index, or a voiced / unvoiced interval. The spectrogram may be any one or a combination of a complex spectrogram, the real part or imaginary part of a complex spectrogram, an amplitude spectrogram, and a phase spectrogram. The spectrogram may also be a spectrogram obtained by converting the scale of a spectrogram such as a mel spectrogram.

[0030] The features of the audio signal may be, for example, quantities extracted from the audio signal using a trainable mathematical model such as a neural network. The trainable model may be, for example, an unsupervised learning model such as a deep generative model, or a model trained by self-supervised learning. The features of the audio signal may be features of a predetermined layer when the audio signal is provided to such a neural network. Note that the trainable model may be trained at any timing as long as it is trained before the features of the audio signal are used. Therefore, the trainable model may be optimized, for example, separately from the training of the student model, or may be trained simultaneously with the training of the student model. The trainable model may be optimized, for example, separately from the training of the teacher model, or may be trained simultaneously with the training of the teacher model.

[0031] <Explanation of Knowledge Distillation Process Using Mathematical Formulas> Here, an example of knowledge distillation process will be explained using mathematical formulas. To facilitate understanding of the knowledge distillation process, learning of the teacher model will be explained using mathematical formulas, although it is well known.

[0032] <<Learning of Teacher Model>> <<<Diffusion Model>>> Diffusion models can be broadly divided into two types: score-based models (SBM) and denoising diffusion probabilistic models (DDPM). Score-based models generate data in the following three steps:

[0033] In the score-based model, first, multiple perturbed distributions are prepared by adding different noises to the original data distribution. Next, the score is estimated for each perturbed distribution. Next, the score-based model generates data by repeating the transition using the Langevin Monte Carlo method using perturbed distributions with gradually decreasing noise levels.

[0034] On the other hand, the denoising diffusion probability model generates data in the following two steps. In the diffusion process of the denoising diffusion probability model, noise is gradually added to the data, converting the data into complete noise. In the dediffusion process of the denoising diffusion probability model, the diffusion process is traced in reverse. That is, in the dediffusion process of the denoising diffusion probability model, noise is gradually removed from complete noise, thereby generating data from noise.

[0035] The score-based model uses a variance-divergence diffusion process that increases noise while keeping the data intact. On the other hand, the denoising diffusion probability model uses a variance-preserving diffusion process that keeps the variance of the model and noise constant. The score-based model and the denoising diffusion probability model can be expressed in a unified manner using the signal-to-noise ratio framework, and the knowledge distillation process can be applied to either model. Below, for simplicity, we will focus on the denoising diffusion probability model.

[0036] In the following, the speech signal or the feature of the speech signal is represented as x. The feature x is data input to the student model in the knowledge distillation process, and is input / output data to the teacher model in the teacher model learning.

[0037] In the denoising diffusion stochastic model, if the number of steps is T, the diffusion process is x ０ →x １ →・・・→x Ｔ The de-diffusion process is expressed as x Ｔ →x Ｔ－１ →・・・→x ０ It is expressed as: x ｔ represents the feature quantity x at step t, where t satisfies the following formula (1).

[0038]

[0039] x ０ represents the data x used for input and output of the denoising diffusion probability model (i.e., data with no noise added). Ｔ represents the complete noise, i.e., the diffusion process is ０Gradually add noise to the Ｔ The despreading process is the process of converting the noise x Ｔ Gradually remove noise from the data x ０ This is the process of generating

[0040] <<<<<Diffusion Process>>>>> Assuming a Markov process, each step of the diffusion process (x ｔ－１ From x ｔ ) transition probability q(x ｔ |x ｔ－１ ) is expressed by the following formula (2).

[0041]

[0042] where α ｔ = 1 - β ｔ The right side of equation (1) is x ｔ The mean is expressed by the following formula (3), and the variance is β ｔ It indicates that the distribution follows a normal distribution represented by I.

[0043]

[0044] Due to the reproducibility of the normal distribution, x ０ x when given ｔ The distribution q(x ｔ |x ０ ) is expressed by the following formula (4).

[0045]

[0046]

[0047] The right side of equation (4) is x ｔ indicates that the mean is expressed by the following formula (6) and the variance follows a normal distribution expressed by the following formula (7).

[0048]

[0049]

[0050] By using the reparameterization trick, the above equation (5) can be replaced with the following equation (8).

[0051]

[0052] Here, the symbol in the following equation (9) represents noise sampled from a normal distribution with a mean of 0 and a variance of I. This noise can be expressed mathematically as the following equation (10).

[0053]

[0054]

[0055] In equation (8), the parameter that needs to be set is β ｔ β ｔ Once is determined, α ｔ = 1 - β ｔ is determined, and the value of the left side of equation (5) is also determined. t may be optimized during training, or a preset constant may be used. t As a method for setting in advance, for example, a cosine schedule may be used.

[0056] <<<<<De-diffusion process>>>>> Each step of the de-diffusion process (x ｔ From x ｔ－１ ) transition probability p θ (x ｔ－１ |x ｔ ) is expressed by the following formula (11).

[0057]

[0058] Here, the right side is x ｔ－１ However, the average μ θ , variance σ ｔ ２ It indicates that I follows a normal distribution. ｔ ２ can be optimized during training or can be a preset constant. ｔ ２ When setting σ ｔ ２ For example, σ ｔ ２ = β ｔ or the following formula (12), or β t and any value between $\tilde{\beta}_t$, where $\tilde{\beta}_t$ refers to the symbol in equation (13) below.

[0059]

[0060]

[0061] Also, σ ｔ ２ If you want to optimize during training, t Optimization may be performed to find the best value between $\tilde{\beta}_t$.

[0062] Also, μ θ is the output of a mathematical model parameterized by θ. The model is modeled, for example, by a deep neural network. The input of the model is, for example, x ｔ , t, s, p are used.

[0063] Here, s represents a speaker information embedding. s may be obtained from a label that identifies the speaker, or may be obtained by a speaker extractor from the input speech or features obtained by converting the input speech. The speaker extractor may be a pre-trained model. The speaker extractor may be optimized simultaneously when training the diffusion model. The speaker extractor may be trained under constraints such that speaker information embeddings extracted from the same speaker's speech are close to each other and speaker information embeddings extracted from different speakers' speech are far apart.

[0064] Furthermore, p represents linguistic information embedding. p may be obtained by a language extractor from the input speech or features obtained by converting the input speech. The language extractor may be a pre-trained model. The language extractor may be optimized simultaneously when training the diffusion model. The language extractor may be trained using, for example, paired data of speech and phonetic posteriorgram (PPG).

[0065] The language extractor may be trained using a self-supervised learning framework such as Hidden Unit Bidirectional Encoder Representations from Transformers (HuBERT) or Large-Scale Model for Cross-Lingual Speech Representation Learning Based on wav2vec 2.0 (XLS-R). The language information embedding uses data from a predetermined layer obtained when the input speech or features obtained by converting the input speech are provided to the language extractor.

[0066] Furthermore, μ θ In the input of μ, p is auxiliary information and does not necessarily have to be used. θ In the input of the above, the case where p is used as auxiliary information will be described as an example.

[0067] By using the reparameterization trick, equation (11) can be replaced by the following equation (14).

[0068]

[0069] Here, z is noise sampled from a normal distribution with a mean of 0 and a variance of I. This noise can be expressed mathematically as the following equation (15).

[0070]

[0071] <<<<Learning of Diffusion Model>>> In the diffusion model, the variational bound of the negative log-likelihood is expressed by the following equation (16): Note that the negative log-likelihood is expressed by the following equation (17).

[0072]

[0073]

[0074] μ θ (x ｔ , t, s, p) can be expressed using a noise estimator by the following equation (18): Note that the noise estimator in equation (18) is the following equation (19).

[0075]

[0076]

[0077] In addition, μ θ In the input of (19), p is auxiliary information and does not necessarily have to be used. If p is not used, p does not have to be used in the input of the noise estimator of equation (19).

[0078] By rewriting equation (16) using equation (18), the following equation (20) is obtained.

[0079]

[0080] Here, w ｔ is α ｔ , $\bar{\alpha}_t$, β ｔ , σ ｔ For example, w ｔ = 1. Note that $\bar{\alpha}_t$ means the following equation (21).

[0081]

[0082] Note that, although the right side of equation (20) shows the case where L2 loss is used, a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0083] Equation (20) indicates that in the despreading process, it is sufficient to train the noise estimator expressed by equation (19) so that the noise given in the spreading process of equation (8) and expressed by equation (9) can be predicted.

[0084] In this way, the teacher model is a speech conversion technology that can learn without using paired data of input speech and output speech, and is a mathematical model that is learned through non-parallel conversion.

[0085] <<<<Conversion Using Diffusion Model>>>> Fig. 3 is a diagram showing Algorithm 1 in the embodiment. Algorithm 1 uses the feature x of the input speech as input information for the diffusion model. ｓｒｃ , speaker information of the target voice sｔｇｔ , the linguistic information of the input speech p ｓｒｃ This is an algorithm for converting the features of input speech into the features of target speech when

[0086] In the following, the superscript src is used when the data is related to the input speech, and the superscript tgt is used when the data is related to the target speech. In Algorithm 1, in order to reduce the number of steps in the conversion, the step sequence expressed by equation (22) in the despreading process is replaced by the subsequence expressed by equation (23).

[0087]

[0088]

[0089] Here, K is an integer equal to or smaller than T. With this substitution, $\alpha_{S_k}$ satisfies the relationship in the following equation (24) when K > 1, and satisfies the relationship in the following equation (25) when K = 1. Note that $\alpha_{S_k}$ refers to the symbol in the following equation (26).

[0090]

[0091]

[0092]

[0093] Also, in the first line of Algorithm 1, the feature x is x ｓｒｃ The initialization is performed using noise x ｓｒｃ Ｔ Alternatively, the data x diffused up to an arbitrary step by Equation (8) may be used for initialization. ｓｒｃ ｔ For example, initialization may be performed using S Ｋ Alternatively, initialization may be performed using data $x^{src}_{S_K}$ diffused up to step $x^{src}_{S_K}$. Here, $x^{src}_{S_K}$ represents the symbol in the following equation (27).

[0094]

[0095] Note that the transformation feature x ｔｇｔ ０For the conversion method from , the conversion feature x ｔｇｔ ０ If the converted feature x is a voice, it may be output as it is. ｔｇｔ ０ However, when the features are converted from speech, the conversion method may be a method of converting features to speech using a neural network such as a neural vocoder, or a method of converting features to speech using a waveform generation algorithm based on signal processing such as the Griffin-Lim algorithm, inverse short-time Fourier transform, or inverse wavelet transform.

[0096] Alternatively, the conversion from features to speech may be performed using a method that combines a neural network and signal processing. When the conversion from features to speech is performed using a neural network such as a neural vocoder, the neural network may be trained in advance or may be optimized simultaneously with training the diffusion model.

[0097] <<Learning of Student Model>> As described above, in the diffusion model, by replacing the step sequence expressed by equation (22) in the dediffusion process during learning with the subsequence expressed by equation (23), the number of steps during conversion can be reduced and the conversion can be sped up. However, this method involves a large trade-off between speed and quality: if the number of steps during conversion is significantly reduced, the quality will be significantly reduced. This is where the knowledge distillation process comes in.

[0098] <<<<Setting Initial Parameters>>> Setting the initial parameters of the student model will be described. The learning parameter of the student model is assumed to be φ. The initial parameter φ during learning of the student model may be set independently of the teacher model, or may be set based on the parameter θ of the teacher model. When set independently of the teacher model, it is set, for example, by randomly sampling a value from a predetermined distribution.

[0099] Furthermore, when the parameters are set based on the parameters of the teacher model, for example, some or all of the parameters φ of the student model are set to be the same as some or all of the parameters θ of the teacher model. Alternatively, when reducing the number of parameters of the student model, the student model may be initialized so that the statistics of the student model match the statistics of the teacher model. Note that the statistics may be any statistics, such as the mean or variance.

[0100] In the following, when data is generated based on a student model having a parameter φ set in this way, the data will be shown with φ added as a subscript.

[0101] <<<<Knowledge Distillation from Teacher Model to Student Model>>> The step sequence of the dediffusion process of the student model is expressed by the following equation (28).

[0102]

[0103] The step sequence S' expressed by equation (28) is a subsequence of the step sequence expressed by equation (22) in the dediffusion process of the teacher model. K' is an integer equal to or less than T. Knowledge distillation from the teacher model to the student model is performed using a loss function that includes at least a loss function based on a diffusion model. As will be described later, knowledge distillation from the teacher model to the student model may be performed using a loss function that includes not only a loss function based on a diffusion model but also a loss function based on adversarial learning.

[0104] <<<<<Loss Function Based on Diffusion Model>>>> The loss function based on the diffusion model will be described. The loss function based on the diffusion model is expressed, for example, by the following equation (29).

[0105]

[0106] Here, x φ is the data (student result) generated by the student model. Specifically, it is obtained by the following equation (30) similar to equation (18).

[0107]

[0108] Here, τ satisfies the relationship of the following equation (31). Also, the symbol $\epsilon_{\phi}$ in the following equation (32) represents the noise estimator of the student model. x τ is expressed as x by equation (8). ０ represents the data diffused by τ steps. τ is expressed mathematically as Equation (33).

[0109]

[0110]

[0111]

[0112] Furthermore, in equation (30), an example is shown in which despreading is performed once using a student model to obtain a student result, but despreading may be performed two or more times using a student model to obtain a student result.

[0113] Also, in formula (29), x' θ is the data (teacher result) obtained by first performing a diffusion process on the student results and then performing a de-diffusion process on the results of the diffusion process using the teacher model. Specifically, it can be obtained, for example, by the following equation (34):

[0114]

[0115] Here, sg represents a process that does not propagate the gradient. Note that although an example in which sg is used to not propagate the gradient has been shown here, the gradient may be propagated without using sg. t satisfies the relationship of the following equation (35). x φ,ｔ is expressed as x by equation (8). φ represents the data diffused for t steps. φ,ｔ is expressed as the formula (36).

[0116]

[0117]

[0118] In addition, in equation (29), c(t) represents a term that is weighted according to t, and for example, a function that decreases the value as t increases is used. Therefore, for example, c(t) = α ｔ is used.

[0119] Although Equation (29) shows the case where the L2 loss is used, a loss function based on any distance criterion such as the L1 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0120] Furthermore, in equation (34), an example has been shown in which the teacher result is obtained by performing despreading once using the teacher model, but the teacher result may be obtained by performing despreading two or more times using the teacher model.

[0121] <<<<<Loss Function Based on Adversarial Learning>>>> The loss function based on adversarial learning is expressed, for example, by the following equations (37) and (38).

[0122]

[0123]

[0124] where D is the classifier, and the real data x ０ and generated data x φ On the other hand, the student model minimizes the equation (38) to identify the generated data x that the classifier D can distinguish from the real data. φ In this way, by optimizing the classifier and the student model under competitive conditions, the student model can generate realistic data that cannot be distinguished even by the strengthened classifier.

[0125] Although Equations (37) and (38) show an example of performing adversarial learning using L2 loss, adversarial learning may be performed based on a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or on a loss function that is a combination of any of these.

[0126] Furthermore, as preprocessing before applying the discriminator D, a process of converting the feature into speech using a waveform generator such as a neural vocoder may be performed. Specifically, when the waveform generator is V and the discriminator applied after applying the waveform generator is D', equations (37) and (38) can be replaced with the following equations (39) and (40), respectively.

[0127]

[0128]

[0129] In equations (39) and (40), the parameter V may remain fixed, or may be optimized simultaneously with the optimization of the student model in equation (40). Note that, when optimizing the student model, equations (39) and (40) may be used simultaneously with equations (37) and (38).

[0130] Although Equations (39) and (40) show an example of performing adversarial learning using L2 loss, adversarial learning may be performed based on a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or on a loss function that is a combination of any of these.

[0131] Furthermore, when performing optimization using equations (37) and (38), the feature matching loss function expressed by the following equation (41) may also be used at the same time.

[0132]

[0133] Here, L represents the number of layers in the network of the classifier D. ｌ represents the number of features in the lth layer. ｌ represents the feature of the lth layer. With this loss function, in the feature space of the classifier D, x ０ and x φ It can bring the two closer together.

[0134] Although Equation (41) shows the case where the L1 loss is used, a loss function based on any distance criterion such as the L2 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0135] Similarly, when optimization is performed using equations (39) and (40), the feature matching loss function expressed by the following equation (42) may also be used at the same time.

[0136]

[0137] Here, L' represents the number of layers in the network of the classifier D'. ｌ represents the number of features in the lth layer. ｌ represents the feature of the lth layer. By this loss function, in the feature space of the classifier D', x ０ and x φ It can bring the two closer together.

[0138] Although Equation (42) shows the case where the L1 loss is used, a loss function based on any distance criterion such as the L2 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0139] <<<<<Overall Loss Function>>>> The overall loss function is expressed, for example, by the following equations (43) to (45).

[0140]

[0141]

[0142]

[0143] The student model, classifier D, and classifier D' are optimized by minimizing L(φ), L(D), and L'(D'), respectively, where λ ｄｉｓｔｉｌｌ , λ ａｄｖ , λ ＦＭ , λ´ ａｄｖ , λ´ ＦＭ is a weighting parameter for each loss function that takes a value greater than or equal to 0, and when it is 0, the associated loss function is ignored.

[0144] However, the overall loss function includes at least a loss function based on the diffusion model. Provided that the overall loss function includes at least a loss function based on the diffusion model, it is not necessary for the overall loss function to include all of the loss functions in Equations (43) to (45). Therefore, the student model may be trained using only some of the loss functions in Equations (43) to (45), provided that the overall loss function includes at least a loss function based on the diffusion model.

[0145] <<<<Transformation Using Student Model>>> Transformation using a student model is similar to transformation using a teacher model, except for differences in parameter values ​​and the number of steps.

[0146] 4 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 is equipped with a control unit 11, which is a control unit including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, all connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0147] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0148] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a knowledge distillation process. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0149] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via a wired or wireless connection.

[0150] The external device is, for example, a device that transmits a signal used in the knowledge distillation process and that is to be converted by the student model. In such a case, the interface unit 12 acquires the signal used in the knowledge distillation process and that is to be converted by the student model by communicating with the device that transmits the signal used in the knowledge distillation process and that is to be converted by the student model.

[0151] The external device is, for example, the conversion device 2. In such a case, the conversion device 2 can execute the knowledge-distilled student model obtained by executing the knowledge distillation process through communication via the interface unit 12.

[0152] The external device is, for example, a device that stores a teacher model. In such a case, the learning device 1 can execute the teacher model in the knowledge distillation process by communicating with the device that stores the teacher model via the interface unit 12.

[0153] The teacher model may be obtained by learning performed by the control unit 11. In this case, the control unit 11 uses the teacher model obtained by learning performed by the control unit 11 itself in the knowledge distillation process to perform knowledge distillation from the teacher model to a student model.

[0154] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the input devices of the interface unit 12 accept input of various information or signals to the learning device 1 via wired or wireless connections. Note that the information or signals do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0155] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output, for example, information input to an input device of the interface unit 12 or information indicated by a signal, as an image or sound.

[0156] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores, for example, various information generated by the operation of the control unit 11. The storage unit 13 may store a teacher model in advance, for example, before knowledge distillation is performed. The storage unit 13 may exist on a cloud, for example.

[0157] An example of the flow of processing executed by the learning device 1 is as described with reference to FIG.

[0158] 5 is a diagram showing an example of the hardware configuration of the conversion device 2 in an embodiment. The conversion device 2 includes a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected via a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0159] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the conversion device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0160] The control unit 21 controls the operation of each functional unit included in the conversion device 2. The control unit 21, for example, executes a conversion process. The control unit 21, for example, acquires information stored in the storage unit 23. Specifically, the process of acquiring information stored in the storage unit 23 is reading.

[0161] The interface unit 22 includes a communication interface for connecting the conversion device 2 to an external device. The interface unit 22 communicates with the external device via a wired or wireless connection. The external device is, for example, a device that is a source of data to be converted by the conversion process executed by the control unit 21.

[0162] The conversion device 2 executes a knowledge-distilled student model, and therefore is a device that executes inference-stage processing in machine learning theory. Therefore, hereinafter, the conversion target of the conversion processing executed by the control unit 21 is referred to as the inference source.

[0163] When the external device is the device that is the source of the inference source, the interface unit 22 acquires the inference source by communicating with the device that is the source of the inference source.

[0164] The external device may be, for example, the learning device 1. In such a case, the conversion device 2 can execute the knowledge-distilled student model obtained by the learning device 1 through communication via the interface unit 22.

[0165] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 22 may be configured as an interface that connects these input devices to the conversion device 2. In this way, the input devices of the interface unit 22 accept input of various information or signals to the conversion device 2 via wired or wireless connections. Note that the various information or signals that can be input to the communication interface of the interface unit 22 do not necessarily have to be input to the communication interface of the interface unit 22, and may instead be input to the input devices of the interface unit 22.

[0166] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the conversion device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 or information indicated by a signal, as an image or sound.

[0167] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the conversion device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0168] 6 is a flowchart showing an example of the flow of processing executed by the conversion device 2 of the embodiment. The control unit 21 of the conversion device 2 acquires an inference source (step S201). Next, the control unit 21 executes a knowledge-distilled student model on the acquired inference source (step S202). That is, the control unit 21 executes a conversion process on the acquired inference source.

[0169] <Experimental Results> An example of the experimental results will be described. <<Experimental Data>> The experimental data used in the experiment will be described. English speech data from 110 speakers was used as the experimental data. To verify the effectiveness of speech conversion from an unknown speaker to another unknown speaker, data from 10 of the 110 speakers was used for evaluation. Data from the remaining 100 speakers was used for training.

[0170] Each speaker's data consisted of approximately 400 sentences of speech data, of which 10 sentences were used for evaluation to verify the effectiveness of speech conversion from unknown speech content to unknown speech content, and the remaining approximately 390 sentences were used for training. Each speech data was sampled at 22.05 kHz. The feature x was an 80-dimensional logarithmic mel spectrogram extracted from a spectrogram obtained by short-time Fourier transform of the speech data with an FFT size of 1024, a shift width of 256, and a window width of 1024.

[0171] <<Model Structure>> The model structure will be explained. The noise estimator of the teacher model was implemented using U-Net consisting of 12 1D convolution layers. This noise estimator is expressed by the following equation (46).

[0172]

[0173] The activation function used was a gated linear unit (GLU), and the weights of the one-dimensional convolutional layers were normalized using weight normalization.

[0174] We will now explain how each input in the noise estimator in Algorithm 1, which is the noise estimator expressed by the following equation (47), is obtained.

[0175]

[0176] t was obtained by embedding using a sinusoidal positional embedding based on trigonometric functions. ｔｇｔ is the target speaker's speech data x using a trained speaker extractor. ｔｇｔ It was obtained by extraction from ｓｒｃ is calculated by using a trained phonetic posterior gram (PPG) estimator to estimate the input speech data x ｓｒｃ Specifically, it was obtained by setting the bottleneck feature of the phoneme posterior probability estimator as p.

[0177] The student model had the same network structure as the teacher model, and was initialized using the teacher model.

[0178] <<Comparative Models>> The comparative models, which are the models to be compared, will now be described. Five comparative models were used in the experiment.

[0179] One of the comparison models was the teacher model A. The teacher model A was a teacher model in which the number of despreading steps K in equation (23) was set to 6.

[0180] The other comparative model was the teacher model B. The teacher model B was a teacher model in which the number of despreading steps K in equation (23) was set to 1.

[0181] The other comparative model was Student Model A. Student Model A is a student model in which the number of steps K' in the de-diffusion process in Equation (28) is set to 1, and the loss function (L ｄｉｓｔｉｌｌ (φ)) was used to perform knowledge distillation.

[0182] The other comparative model was Student Model B. Student Model B is a student model in which the number of steps in the de-diffusion process, K', is set to 1 in Equation (28), and the loss function (L' ａｄｖ (φ), L' ａｄｖ (D'), L' ＦＭ (φ)) was used to perform knowledge distillation.

[0183] The other comparative model was Student Model C. Student Model C is a student model in which the number of steps K' in the de-diffusion process in Equation (28) is set to 1, and the loss function (L ｄｉｓｔｉｌｌ (φ)) and the loss function based on adversarial learning (L´ ａｄｖ (φ), L' ａｄｖ (D'), L' ＦＭ (φ)) and knowledge distillation was performed.

[0184] Due to the difference in the number of dediffusion steps, the conversion speeds of teacher model B, student model A, student model B, and student model C are equivalent, but the conversion speed of teacher model A is six times slower. Note that teacher model A and teacher model B are mathematical models that have not undergone knowledge distillation processing, and student model A, student model B, and student model C are mathematical models that have undergone knowledge distillation processing.

[0185] <<Evaluation Indicators>> The evaluation indices used in the experiment will be explained below. Six evaluation indices were used in the experiment.

[0186] One of the evaluation indices was qMOS, a Mean Opinion Score (MOS) obtained from subjective evaluation experiments on voice quality. It is rated on a five-point scale (5: Excellent, 4: Good, 3: Fair, 2: Poor, 1: Bad), with higher values ​​indicating better quality.

[0187] Another evaluation index was the Mean Opinion Score (sMOS), which is the result of subjective evaluation experiments on speaker similarity. It is rated on a four-point scale (4: Same (sure), 3: Same (not sure), 2: Different (not sure), 1: Different (sure)), and the higher the value, the closer the converted speech is to the target speech in terms of speaker similarity.

[0188] Another evaluation index was UTMOS (UTokyo-SaruLab Mean Opinion Score). This is a Mean Opinion Score (MOS) predicted by a MOS predictor that predicts speech quality. Specifically, the MOS predictor was trained by using speech data with MOS assigned as training data and learning the relationship between the speech data and MOS using a deep learning method. The training data was collected for the purpose of evaluating the quality of speech synthesis and speech conversion, and can predict MOS, which relates to the quality of speech synthesis and conversion. MOS training data takes values ​​from 1 to 5, with higher values ​​indicating better quality.

[0189] Another evaluation index was DNSMOS (Deep Noise Suppression Mean Opinion Score). This is a Mean Opinion Score (MOS) predicted by a MOS predictor that predicts speech quality. Specifically, the MOS predictor was trained by using speech data to which MOS was assigned as training data and learning the relationship between the speech data and MOS using a deep learning method. The training data was collected for the purpose of evaluating the quality of speech noise suppression, and can predict MOS related to the quality of speech noise suppression. The MOS training data takes values ​​from 1 to 5, with higher values ​​indicating better quality.

[0190] Another evaluation index was the character error rate (CER), which is the probability that the speech recognizer will fail to recognize the converted speech. A smaller CER indicates that the speech recognizer is able to convert the converted speech into speech that can be correctly recognized.

[0191] Another evaluation index was SVA, which stands for Speaker Verification Accuracy. SVA is the probability that the speaker verifying device determines that the converted speech and the target speech are from the same speaker. A higher SVA value indicates that the converted speech is closer to the target speech in terms of speaker identity.

[0192] 7 is a diagram showing an example of the results of an experiment in an embodiment. Comparing teacher model A and teacher model B, Fig. 7 shows that teacher model B has significantly lower performance in multiple indices (qMOS, UTMOS) than teacher model A. This shows that simply reducing the number of steps can increase speed, but results in a significant deterioration in performance.

[0193] In contrast, Figure 7 shows that student model A, which underwent knowledge distillation based on a loss function based on a diffusion model, is effective in improving UTMOS, and obtains a score equivalent to that of teacher model A.

[0194] On the other hand, student model B, which underwent knowledge distillation based on a loss function based on adversarial learning, is effective in improving qMOS and DNSMOS, and Figure 7 shows that it achieved a score equivalent to that of teacher model A in DNSMOS.

[0195] Figure 7 shows that Student Model C, which combines Student Model A and Student Model B, is able to utilize the strengths of both models, and outperforms Teacher Model A and Teacher Model B in all scores. In particular, Figure 7 shows that Student Model C has the same conversion speed as Teacher Model B, but is six times faster than Teacher Model A.

[0196] In this way, FIG. 7 shows that knowledge distillation from a teacher model to a student model by knowledge distillation processing enables an increase in conversion speed while maintaining speech quality and speaker similarity.

[0197] The conversion system 100 configured in this manner includes the learning device 1 that executes the knowledge distillation process. Therefore, as described in <Effects of the knowledge distillation process>, the conversion speed can be increased.

[0198] Furthermore, the learning device 1 configured in this manner executes knowledge distillation processing, which makes it possible to increase the conversion speed, as described in <Effects of Knowledge Distillation Processing>.

[0199] Furthermore, the conversion device 2 configured in this manner performs conversion using a knowledge-distilled student model obtained by the knowledge distillation process, thereby enabling the conversion speed to be increased.

[0200] (Variations) Although the explanation so far has been given using voice conversion as an example, the knowledge distillation process is not necessarily applicable only to student models that perform voice conversion, but can be applied to any signal. In other words, for any signal, the knowledge distillation process can be applied to a student model that performs conversion on that signal. The signal may be, for example, an acoustic signal, a biological signal, or another signal acquired by any sensor.

[0201] Therefore, the knowledge distillation process is a process of performing knowledge distillation using, as a teacher model, a mathematical model that performs signal transformation using a diffusion model and that has undergone non-parallel learning. That is, the teacher model is a mathematical model that performs signal transformation using a diffusion model and that has undergone non-parallel learning. Note that non-parallel learning is learning that can be performed in learning mapping from a source to a destination without requiring parallel data of the source and the destination.

[0202] In this case, the student model is a mathematical model that converts signals using a diffusion model and has fewer steps in the dediffusion process than the teacher model.

[0203] In the above knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model.

[0204] The conversion process executed by the conversion device 2 is a process of converting the conversion target using the knowledge-distilled student model obtained by the learning device 1.

[0205] <Another Example of Knowledge Distillation> So far, as an example of knowledge distillation, an example using a loss function including a diffusion loss (hereinafter referred to as "first knowledge distillation") has been described. That is, we have described knowledge distillation in which the student model is updated so as to reduce the difference between the result of transforming a signal in training data using a student model and the result of transforming that result using a teacher model. However, this is just one example.

[0206] The knowledge distillation may be, for example, second knowledge distillation. The second knowledge distillation is knowledge distillation that updates a student model so as to reduce the difference between the result of transforming a signal in training data using a student model and the result of transforming the same signal in training data using a teacher model. In this case, optimization is performed so that the input-output correspondence between the student model and the teacher model matches.

[0207] In this way, in knowledge distillation, the student model may be updated to reduce the difference between the result of transforming a signal by the student model and the result of transforming the signal by the teacher model. Thus, for example, in knowledge distillation, the student model may be updated to reduce the difference between the result of transforming a signal by the student model and the result of transforming that signal or its result by the teacher model.

[0208] In the first knowledge distillation, the teacher model is applied again to the result of conversion by the student model. Therefore, the input / output correspondence between the student model and the teacher model does not necessarily need to match. Therefore, the first knowledge distillation is easier to handle the case where a destination signal corresponding to a source signal is not uniquely determined than the second knowledge distillation.

[0209] We will explain the ease of adaptation. When a destination signal corresponding to a source signal is not uniquely determined, there are multiple solutions for the destination signal. In this case, in the second knowledge distillation, the student model must generate a solution that matches the teacher model, and strong constraints are imposed. On the other hand, in the first knowledge distillation, the student model may generate a solution that does not match the teacher model, and strong constraints are not imposed. In this way, ease of adaptation means that the constraints imposed in the first knowledge distillation are weaker than those in the second knowledge distillation.

[0210] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0211] Note that all or part of the functions of the learning device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media (ROM, RAM, etc.) such as magnetic disks, magneto-optical disks, optical disks (CD-ROM, DVD-ROM, etc.), semiconductor memories (volatile memory, non-volatile memory, etc.), and storage devices such as hard disks built into computer systems. The program may also be transmitted via telecommunications lines.

[0212] The control unit 21 is an example of a conversion unit.

[0213] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0214] REFERENCE SIGNS LIST 100...conversion system, 1...learning device, 2...conversion device, 11...control unit, 12...interface unit, 13...storage unit, 21...control unit, 22...interface unit, 23...storage unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. A learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation; wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

2. A conversion device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation, wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model, and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model.

3. A learning method executed by a learning device, comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model, and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model, the learning method comprising: a knowledge distillation step in which the control unit performs the knowledge distillation.

4. A program for causing a computer to function as the learning device according to claim 1.

Images