Learning device, conversion device, learning method, and program

Abstract

This learning device comprises a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that performs signal conversion using a diffusion model and has been trained by non-parallel conversion. A student model in the knowledge distillation is a mathematical model that performs signal conversion using a diffusion model and has fewer steps in a despreading process than the teacher model. In the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal conversion by the student model and the result of signal conversion by the teacher model.

Description

Learning device, conversion device, learning method and program

[0001] The present invention relates to a learning device, a conversion device, a learning method, and a program.

[0002] Speech conversion is a technology that converts only non-linguistic and paralinguistic information from input speech while preserving the linguistic information. Speech conversion can be applied to a variety of technologies, including speaker characteristics, emotion, and pronunciation conversion, as well as speech enhancement and speech assistance. Training methods for speech conversion can be broadly divided into parallel speech conversion and non-parallel speech conversion.

[0003] Parallel speech conversion is a training method that uses paired data of input and output speech as training data. Parallel speech conversion has the advantage of being easy to train because it can explicitly obtain the correct target speech for the converted speech. However, training requires collecting paired data of input and output speech with the same speech content, which has the disadvantage of high training data collection costs.

[0004] Non-parallel speech conversion has been proposed as a method to overcome the above problems. Non-parallel speech conversion is a speech conversion technology that allows training without using paired data of input speech and output speech. This eliminates the need to collect paired data of input speech and output speech with the same speech content when training, making it possible to reduce the cost of collecting training data.

[0005] On the other hand, unlike parallel speech conversion, it is not possible to explicitly obtain the correct target speech for the converted speech, so training is not easy.

[0006] As a method for solving this problem, non-parallel speech conversion using a diffusion model has been proposed (see Non-Patent Document 1). In the diffusion model, the process of converting data into latent variables (noise) (diffusion process) is realized by gradually adding noise, while the process of converting the latent variables (noise) into data (de-diffusion process) is realized by gradually removing noise.

[0007] Since training is performed to maximize the logarithmic likelihood when diffusion and de-diffusion are performed on each piece of speech data that is not paired, training is possible even if there is no paired data of input speech and output speech with the same speech content.In addition, by performing repeated calculations, the diffusion model can express complex functions that are difficult to express in a single calculation, making it possible to perform high-quality speech conversion.

[0008] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo, and Shogo Seki. VoiceGrad: Non-parallel any-to-many voice conversion with annealed Langevin dynamics. IEEE / ACM Trans. Audio Speech Lang. Process., vol. 32, pp. 2213-2226, 2024.Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Yuto Kondo. FastVoiceGrad: One-step Diffusion-Based Voice Conversion with Adversarial Conditional Diffusion Distillation, Interspeech, 2024.

[0009] However, the calculation time increases in proportion to the number of iterative calculations, resulting in slow conversion speed.

[0010] In view of the above circumstances, an object of the present invention is to provide a technique for increasing the conversion speed.

[0011] One aspect of the present invention is a learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms a signal using a diffusion model and that has been learned through non-parallel transformation; a student model in the knowledge distillation is a mathematical model that transforms a signal using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and the student model is updated in the knowledge distillation so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

[0012] One aspect of the present invention is a conversion device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms a signal using a diffusion model and that has been learned by non-parallel transformation; wherein the student model in the knowledge distillation is a mathematical model that transforms a signal using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and wherein, in the knowledge distillation, the student model is updated so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model; and a conversion unit that performs conversion using a knowledge-distilled student model obtained by a learning device.

[0013] One aspect of the present invention is a learning method executed by a learning device, the learning method including a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model, and the student model is updated in the knowledge distillation so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model, the learning method including a knowledge distillation step in which the control unit performs the knowledge distillation.

[0014] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0015] One aspect of the present invention is a learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; and a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of calculations than the teacher model.

[0016] One aspect of the present invention is a conversion device including: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; and a conversion unit that performs conversion using a knowledge-distilled student model obtained by a learning device, wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model.

[0017] One aspect of the present invention is a learning method executed by a learning device, the learning method including: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; and a student model in the knowledge distillation that is a mathematical model that transforms signals using a diffusion model and that has fewer operations than the teacher model, the learning method including a knowledge distillation step in which the control unit performs the knowledge distillation.

[0018] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0019] The present invention makes it possible to increase the conversion speed.

[0020] 1 is an explanatory diagram illustrating a conversion system according to an embodiment. A flowchart illustrating an example of a processing flow executed by a control unit according to an embodiment. A diagram illustrating algorithm 1 according to an embodiment. A diagram illustrating an example of a hardware configuration of a learning device according to an embodiment. A diagram illustrating an example of a hardware configuration of a conversion device according to an embodiment. A flowchart illustrating an example of a processing flow executed by a conversion device according to an embodiment. A diagram illustrating an example of an experimental result according to an embodiment. A diagram illustrating an example of an experimental result according to a modified example. A flowchart illustrating an example of a processing flow executed by a control unit according to a modified example. A flowchart illustrating an example of a processing flow executed by a conversion device according to a modified example. An explanatory diagram illustrating a generation system according to a modified example.

[0021] 1 is an explanatory diagram illustrating a conversion system 100 according to an embodiment. The conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 includes a control unit 11 including a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus. The control unit 11 executes knowledge distillation processing.

[0022] The knowledge distillation process is a process of performing knowledge distillation using a mathematical model that performs speech conversion using a diffusion model and that has been trained by non-parallel speech conversion as a teacher model. Therefore, the teacher model used in this knowledge distillation is a mathematical model that performs speech conversion using a diffusion model and that has been trained by non-parallel speech conversion. Furthermore, the student model in this knowledge distillation is a mathematical model that performs speech conversion using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model.

[0023] The conversion device 2 performs a conversion process, which is a process of converting speech using the student model obtained by the learning device 1 after knowledge distillation.

[0024] The term "knowledge distillation completed" means that knowledge distillation has been performed until a predetermined condition for terminating knowledge distillation (hereinafter referred to as "knowledge distillation termination condition") is satisfied.

[0025] <Effects of knowledge distillation processing> A student model has fewer steps in the dediffusion process than a teacher model. Therefore, a knowledge-distilled student model can perform speech conversion with fewer steps than a teacher model, with almost the same accuracy as that of conversion using a teacher model. Therefore, a knowledge-distilled student model can increase the conversion speed.

[0026] 2 is a flowchart showing an example of the flow of processing executed by the control unit 11 in an embodiment. More specifically, it is a flowchart showing an example of the flow of processing executed in the knowledge distillation process in an embodiment. The control unit 11 acquires a conversion target that is a conversion target by a student model (hereinafter referred to as a "student conversion target") from the training data (step S101). Specifically, the student conversion target is one or more voice signals (voice data) or one or more feature quantities of a voice signal. Next, the control unit 11 performs voice conversion on the student conversion target using the student model (step S102).

[0027] Next, the control unit 11 performs voice conversion on the result of the voice conversion performed in step S102 (hereinafter referred to as the "student result") using the teacher model (step S103). Hereinafter, the result of voice conversion by the teacher model will be referred to as the teacher result. Note that since the teacher model is a diffusion model, in step S103 the teacher model first performs a diffusion process on the student result, and then performs a de-diffusion process on the result of the diffusion process. The result of this de-diffusion process is the teacher result.

[0028] The control unit 11 updates the student model in accordance with a predetermined rule (step S104).

[0029] After step S104, the control unit 11 determines whether a knowledge distillation termination condition is satisfied (step S105). The knowledge distillation termination condition may be any condition related to the termination of knowledge distillation, such as a condition that the student model has been updated a predetermined number of times, a condition that the change in the student model due to the update is smaller than a predetermined change, or a condition that the score satisfies a predetermined condition related to the value of a predetermined index for evaluating the performance of the student model. An example of the condition that the score satisfies a predetermined condition related to the value of a predetermined index for evaluating the performance of the student model is a condition that the difference between the student result and the teacher result is smaller than a predetermined difference.

[0030] If the knowledge distillation termination condition is satisfied (step S105: YES), the knowledge distillation process ends. On the other hand, if the knowledge distillation termination condition is not satisfied (step S105: NO), the process returns to step S101 after step S105.

[0031] <Regarding Loss Function> In the above knowledge distillation, any loss function may be used as long as knowledge distillation from a teacher model to a student model is performed. For example, the loss function may include a loss function based on a diffusion model used by the student model (hereinafter referred to as "diffusion loss"). Specifically, the diffusion loss is a value indicating the difference between the teacher result and the diffusion result. Furthermore, the loss function used in the above knowledge distillation may include a loss function based on adversarial learning (hereinafter referred to as "adversarial loss") in addition to the diffusion loss.

[0032] <Regarding Initial Values ​​of Student Model> Any value may be used as the initial value of the parameter of the student model in the above knowledge distillation as long as knowledge distillation is performed from the teacher model to the student model. For example, the value of the parameter of the teacher model may be used.

[0033] <Regarding Audio Signal Features> The audio signal features described above will now be explained. The audio signal features may be quantities extracted using a predetermined function. The predetermined function may be, for example, a transformation from a waveform to a spectrogram based on basis functions such as a short-time Fourier transform or a wavelet transform. The predetermined function may be, for example, a transformation into a cepstrum obtained by further performing Fourier analysis on the spectrogram, or a transformation into acoustic features such as a fundamental frequency, an asynchrony index, or a voiced / unvoiced interval. The spectrogram may be any one or a combination of a complex spectrogram, the real part or imaginary part of a complex spectrogram, an amplitude spectrogram, and a phase spectrogram. The spectrogram may also be a spectrogram obtained by converting the scale of a spectrogram such as a mel spectrogram.

[0034] The features of the audio signal may be, for example, quantities extracted from the audio signal using a trainable mathematical model such as a neural network. The trainable model may be, for example, an unsupervised learning model such as a deep generative model, or a model trained by self-supervised learning. The features of the audio signal may be features of a predetermined layer when the audio signal is provided to such a neural network. Note that the trainable model may be trained at any timing as long as it is trained before the features of the audio signal are used. Therefore, the trainable model may be optimized, for example, separately from the training of the student model, or may be trained simultaneously with the training of the student model. The trainable model may be optimized, for example, separately from the training of the teacher model, or may be trained simultaneously with the training of the teacher model.

[0035] <Explanation of Knowledge Distillation Process Using Mathematical Formulas> Here, an example of knowledge distillation process will be explained using mathematical formulas. To facilitate understanding of the knowledge distillation process, learning of the teacher model will be explained using mathematical formulas, although it is well known.

[0036] <<Learning of Teacher Model>> <<<Diffusion Model>>> Diffusion models can be broadly divided into two types: score-based models (SBM) and denoising diffusion probabilistic models (DDPM). Score-based models generate data in the following three steps:

[0037] In the score-based model, first, multiple perturbed distributions are prepared by adding different noises to the original data distribution. Next, the score is estimated for each perturbed distribution. Next, the score-based model generates data by repeating the transition using the Langevin Monte Carlo method using perturbed distributions with gradually decreasing noise levels.

[0038] On the other hand, the denoising diffusion probability model generates data in the following two steps. In the diffusion process of the denoising diffusion probability model, noise is gradually added to the data, converting the data into complete noise. In the dediffusion process of the denoising diffusion probability model, the diffusion process is traced in reverse. That is, in the dediffusion process of the denoising diffusion probability model, noise is gradually removed from complete noise, thereby generating data from noise.

[0039] The score-based model uses a variance-divergence diffusion process that increases noise while keeping the data intact. On the other hand, the denoising diffusion probability model uses a variance-preserving diffusion process that keeps the variance of the model and noise constant. The score-based model and the denoising diffusion probability model can be expressed in a unified manner using the signal-to-noise ratio framework, and the knowledge distillation process can be applied to either model. Below, for simplicity, we will focus on the denoising diffusion probability model.

[0040] In the following, the speech signal or the feature of the speech signal is represented as x. The feature x is data input to the student model in the knowledge distillation process, and is input / output data to the teacher model in the teacher model learning.

[0041] In the denoising diffusion stochastic model, if the number of steps is T, the diffusion process is x ０ →x １ →・・・→x Ｔ The de-diffusion process is expressed as x Ｔ →x Ｔ－１ →・・・→x ０ It is expressed as: x ｔ represents the feature quantity x at step t, where t satisfies the following formula (1).

[0042]

[0043] x ０ represents the data x used for input and output of the denoising diffusion probability model (i.e., data with no noise added). Ｔ represents the complete noise, i.e., the diffusion process is ０ Gradually add noise to the Ｔ The despreading process is the process of converting the noise x Ｔ Gradually remove noise from the data x ０ This is the process of generating

[0044] <<<<<Diffusion Process>>>>> Assuming a Markov process, each step of the diffusion process (x ｔ－１ From x ｔ ) transition probability q(x ｔ |x ｔ－１ ) is expressed by the following formula (2).

[0045]

[0046] where α ｔ = 1 - β ｔ and the right side of equation (1) is x ｔ The mean is expressed by the following formula (3), and the variance is β ｔ It indicates that the distribution follows a normal distribution represented by I.

[0047]

[0048] Due to the reproducibility of the normal distribution, x ０ x when given ｔ The distribution q(x ｔ |x ０) is expressed by the following formula (4).

[0049]

[0050]

[0051] The right side of equation (4) is x ｔ indicates that the mean is expressed by the following formula (6) and the variance follows a normal distribution expressed by the following formula (7).

[0052]

[0053]

[0054] By using the reparameterization trick, the above equation (5) can be replaced with the following equation (8).

[0055]

[0056] Here, the symbol in the following equation (9) represents noise sampled from a normal distribution with a mean of 0 and a variance of I. This noise can be expressed mathematically as the following equation (10).

[0057]

[0058]

[0059] In equation (8), the parameter that needs to be set is β ｔ β ｔ Once is determined, α ｔ = 1 - β ｔ is determined, and the value of the left side of equation (5) is also determined. t may be optimized during training, or a preset constant may be used. t As a method for setting in advance, for example, a cosine schedule may be used.

[0060] <<<<<De-diffusion process>>>>> Each step of the de-diffusion process (x ｔ From x ｔ－１ ) transition probability p θ (x ｔ－１ |x ｔ ) is expressed by the following formula (11).

[0061]

[0062] Here, the right side is x ｔ－１ However, the average μ θ , variance σ ｔ ２ It indicates that I follows a normal distribution. ｔ ２ can be optimized during training or can be a preset constant. ｔ ２ When setting σ ｔ ２ For example, σ ｔ ２ = β ｔ or the following formula (12), or β t and any value between $\tilde{\beta}_t$, where $\tilde{\beta}_t$ refers to the symbol in equation (13) below.

[0063]

[0064]

[0065] Also, σ ｔ ２ If you want to optimize during training, t Optimization may be performed to find the best value between $\tilde{\beta}_t$.

[0066] Also, μ θ is the output of a mathematical model parameterized by θ. The model is modeled, for example, by a deep neural network. The input of the model is, for example, x ｔ , t, s, p are used.

[0067] Here, s represents a speaker information embedding. s may be obtained from a label that identifies the speaker, or may be obtained by a speaker extractor from the input speech or features obtained by converting the input speech. The speaker extractor may be a pre-trained model. The speaker extractor may be optimized simultaneously when training the diffusion model. The speaker extractor may be trained under constraints such that speaker information embeddings extracted from the same speaker's speech are close to each other and speaker information embeddings extracted from different speakers' speech are far apart.

[0068] Furthermore, p represents linguistic information embedding. p may be obtained by a language extractor from the input speech or features obtained by converting the input speech. The language extractor may be a pre-trained model. The language extractor may be optimized simultaneously when training the diffusion model. The language extractor may be trained using, for example, paired data of speech and phonetic posteriorgram (PPG).

[0069] The language extractor may be trained using a self-supervised learning framework such as Hidden Unit Bidirectional Encoder Representations from Transformers (HuBERT) or Large-Scale Model for Cross-Lingual Speech Representation Learning Based on wav2vec 2.0 (XLS-R). The language information embedding uses data from a predetermined layer obtained when the input speech or features obtained by converting the input speech are provided to the language extractor.

[0070] Furthermore, μ θ In the input of μ, p is auxiliary information and does not necessarily have to be used. θ In the input of the above, the case where p is used as auxiliary information will be described as an example.

[0071] By using the reparameterization trick, equation (11) can be replaced by the following equation (14).

[0072]

[0073] Here, z is noise sampled from a normal distribution with a mean of 0 and a variance of I. This noise can be expressed mathematically as the following equation (15).

[0074]

[0075] <<<<Learning of Diffusion Model>>> In the diffusion model, the variational bound of the negative log-likelihood is expressed by the following equation (16): Note that the negative log-likelihood is expressed by the following equation (17).

[0076]

[0077]

[0078] μ θ (x ｔ , t, s, p) can be expressed using a noise estimator by the following equation (18): Note that the noise estimator in equation (18) is the following equation (19).

[0079]

[0080]

[0081] In addition, μ θ In the input of (19), p is auxiliary information and does not necessarily have to be used. If p is not used, p does not have to be used in the input of the noise estimator of equation (19).

[0082] By rewriting equation (16) using equation (18), the following equation (20) is obtained.

[0083]

[0084] Here, w ｔ is α ｔ , $\bar{\alpha}_t$, β ｔ , σ ｔ For example, w ｔ = 1. Note that $\bar{\alpha}_t$ means the following equation (21).

[0085]

[0086] Note that, although the right side of equation (20) shows the case where L2 loss is used, a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0087] Equation (20) indicates that in the despreading process, it is sufficient to train the noise estimator expressed by equation (19) so that the noise given in the spreading process of equation (8) and expressed by equation (9) can be predicted.

[0088] In this way, the teacher model is a speech conversion technology that can learn without using paired data of input speech and output speech, and is a mathematical model that is learned through non-parallel conversion.

[0089] <<<<Conversion Using Diffusion Model>>>> Fig. 3 is a diagram showing Algorithm 1 in the embodiment. Algorithm 1 uses the feature x of the input speech as input information for the diffusion model. ｓｒｃ , speaker information of the target voice s ｔｇｔ , the linguistic information of the input speech p ｓｒｃ This is an algorithm for converting the features of input speech into the features of target speech when

[0090] In the following, the superscript src is used when the data is related to the input speech, and the superscript tgt is used when the data is related to the target speech. In Algorithm 1, in order to reduce the number of steps in the conversion, the step sequence expressed by equation (22) in the despreading process is replaced by the subsequence expressed by equation (23).

[0091]

[0092]

[0093] Here, K is an integer equal to or smaller than T. With this substitution, $\alpha_{S_k}$ satisfies the relationship in the following equation (24) when K > 1, and satisfies the relationship in the following equation (25) when K = 1. Note that $\alpha_{S_k}$ refers to the symbol in the following equation (26).

[0094]

[0095]

[0096]

[0097] Also, in the first line of Algorithm 1, the feature x is x ｓｒｃ The initialization is performed using noise x ｓｒｃ Ｔ Alternatively, the data x diffused up to an arbitrary step by Equation (8) may be used for initialization. ｓｒｃ ｔ For example, initialization may be performed using S Ｋ Alternatively, initialization may be performed using data $x^{src}_{S_K}$ diffused up to step $x^{src}_{S_K}$. Here, $x^{src}_{S_K}$ represents the symbol in the following equation (27).

[0098]

[0099] Note that the transformation feature x ｔｇｔ ０ For the conversion method from , the conversion feature x ｔｇｔ ０ If the converted feature x is a voice, it may be output as it is. ｔｇｔ ０ However, when the features are converted from speech, the conversion method may be a method of converting features to speech using a neural network such as a neural vocoder, or a method of converting features to speech using a waveform generation algorithm based on signal processing such as the Griffin-Lim algorithm, inverse short-time Fourier transform, or inverse wavelet transform.

[0100] Alternatively, the conversion from features to speech may be performed using a method that combines a neural network and signal processing. When the conversion from features to speech is performed using a neural network such as a neural vocoder, the neural network may be trained in advance or may be optimized simultaneously with training the diffusion model.

[0101] <<Learning of Student Model>> As described above, in the diffusion model, by replacing the step sequence expressed by equation (22) in the dediffusion process during learning with the subsequence expressed by equation (23), the number of steps during conversion can be reduced and the conversion can be sped up. However, this method involves a large trade-off between speed and quality: if the number of steps during conversion is significantly reduced, the quality will be significantly reduced. This is where the knowledge distillation process comes in.

[0102] <<<<Setting Initial Parameters>>> Setting the initial parameters of the student model will be described. The learning parameter of the student model is assumed to be φ. The initial parameter φ during learning of the student model may be set independently of the teacher model, or may be set based on the parameter θ of the teacher model. When set independently of the teacher model, it is set, for example, by randomly sampling a value from a predetermined distribution.

[0103] Furthermore, when the parameters are set based on the parameters of the teacher model, for example, some or all of the parameters φ of the student model are set to be the same as some or all of the parameters θ of the teacher model. Alternatively, when reducing the number of parameters of the student model, the student model may be initialized so that the statistics of the student model match the statistics of the teacher model. Note that the statistics may be any statistics, such as the mean or variance.

[0104] In the following, when data is generated based on a student model having a parameter φ set in this way, the data will be shown with φ added as a subscript.

[0105] <<<<Knowledge Distillation from Teacher Model to Student Model>>> The step sequence of the dediffusion process of the student model is expressed by the following equation (28).

[0106]

[0107] The step sequence S' expressed by equation (28) is a subsequence of the step sequence expressed by equation (22) in the dediffusion process of the teacher model. K' is an integer equal to or less than T. Knowledge distillation from the teacher model to the student model is performed using a loss function that includes at least a loss function based on a diffusion model. As will be described later, knowledge distillation from the teacher model to the student model may be performed using a loss function that includes not only a loss function based on a diffusion model but also a loss function based on adversarial learning.

[0108] <<<<<Loss Function Based on Diffusion Model>>>> The loss function based on the diffusion model will be described. The loss function based on the diffusion model is expressed, for example, by the following equation (29).

[0109]

[0110] Here, x φ is the data (student result) generated by the student model. Specifically, it is obtained by the following equation (30) similar to equation (18).

[0111]

[0112] Here, τ satisfies the relationship of the following equation (31). Also, the symbol $\epsilon_{\phi}$ in the following equation (32) represents the noise estimator of the student model. x τ is expressed as x by equation (8). ０ represents the data that has been diffused by τ steps. τ is expressed mathematically as Equation (33).

[0113]

[0114]

[0115]

[0116] Furthermore, in equation (30), an example is shown in which despreading is performed once using a student model to obtain a student result, but despreading may be performed two or more times using a student model to obtain a student result.

[0117] Also, in formula (29), x' θis the data (teacher result) obtained by first performing a diffusion process on the student results and then performing a de-diffusion process on the results of the diffusion process using the teacher model. Specifically, it can be obtained, for example, by the following equation (34).

[0118]

[0119] Here, sg represents a process that does not propagate the gradient. Although an example in which sg is used to not propagate the gradient is shown here, the gradient may be propagated without using sg. t satisfies the relationship of the following equation (35). x φ,ｔ is expressed as x by equation (8). φ represents the data diffused for t steps. φ,ｔ is expressed as the formula (36).

[0120]

[0121]

[0122] In addition, in equation (29), c(t) represents a term that is weighted according to t, and for example, a function that decreases the value as t increases is used. Therefore, for example, c(t) = α ｔ is used.

[0123] Although Equation (29) shows the case where the L2 loss is used, a loss function based on any distance criterion such as the L1 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0124] Furthermore, in equation (34), an example has been shown in which the teacher result is obtained by performing despreading once using the teacher model, but the teacher result may be obtained by performing despreading two or more times using the teacher model.

[0125] <<<<<Loss Function Based on Adversarial Learning>>>> The loss function based on adversarial learning is expressed, for example, by the following equations (37) and (38).

[0126]

[0127]

[0128] where D is the classifier, and the real data x ０ and generated data x φ On the other hand, the student model minimizes the equation (38) to identify the generated data x that the classifier D can distinguish from the real data. φ In this way, by optimizing the classifier and the student model under competitive conditions, the student model can generate realistic data that cannot be distinguished even by the strengthened classifier.

[0129] Although Equations (37) and (38) show an example of performing adversarial learning using L2 loss, adversarial learning may be performed based on a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or on a loss function that is a combination of any of these.

[0130] Furthermore, as preprocessing before applying the discriminator D, a process of converting the feature into speech using a waveform generator such as a neural vocoder may be performed. Specifically, when the waveform generator is V and the discriminator applied after applying the waveform generator is D', equations (37) and (38) can be replaced with the following equations (39) and (40), respectively.

[0131]

[0132]

[0133] In equations (39) and (40), the parameter V may remain fixed, or may be optimized simultaneously with the optimization of the student model in equation (40). Note that, when optimizing the student model, equations (39) and (40) may be used simultaneously with equations (37) and (38).

[0134] Although Equations (39) and (40) show an example of performing adversarial learning using L2 loss, adversarial learning may be performed based on a loss function based on any distance criterion such as L1 loss, hinge loss, cross-entropy loss, or Wasserstein loss, or on a loss function that is a combination of any of these.

[0135] Furthermore, when performing optimization using equations (37) and (38), the feature matching loss function expressed by the following equation (41) may also be used at the same time.

[0136]

[0137] Here, L represents the number of layers in the network of the classifier D. ｌ represents the number of features in the lth layer. ｌ represents the feature of the lth layer. With this loss function, in the feature space of the classifier D, x ０ and x φ It can bring the two closer together.

[0138] Although Equation (41) shows the case where the L1 loss is used, a loss function based on any distance criterion such as the L2 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0139] Similarly, when optimization is performed using equations (39) and (40), the feature matching loss function expressed by the following equation (42) may also be used at the same time.

[0140]

[0141] Here, L' represents the number of layers in the network of the classifier D'. ｌ represents the number of features in the lth layer. ｌ represents the feature of the lth layer. By this loss function, in the feature space of the classifier D', x ０ and x φ It can bring the two closer together.

[0142] Although Equation (42) shows the case where the L1 loss is used, a loss function based on any distance criterion such as the L2 loss, the hinge loss, the cross-entropy loss, or the Wasserstein loss, or a loss function that is a combination of any of these may also be used.

[0143] <<<<<Overall Loss Function>>>> The overall loss function is expressed, for example, by the following equations (43) to (45).

[0144]

[0145]

[0146]

[0147] The student model, classifier D, and classifier D' are optimized by minimizing L(φ), L(D), and L'(D'), respectively, where λ ｄｉｓｔｉｌｌ , λ ａｄｖ , λ ＦＭ , λ´ ａｄｖ , λ´ ＦＭ is a weighting parameter for each loss function that takes a value greater than or equal to 0, and when it is 0, the associated loss function is ignored.

[0148] However, the overall loss function includes at least a loss function based on the diffusion model. Provided that the overall loss function includes at least a loss function based on the diffusion model, it is not necessary for the overall loss function to include all of the loss functions in Equations (43) to (45). Therefore, the student model may be trained using only some of the loss functions in Equations (43) to (45), provided that the overall loss function includes at least a loss function based on the diffusion model.

[0149] <<<<Transformation Using Student Model>>> Transformation using a student model is similar to transformation using a teacher model, except for differences in parameter values ​​and the number of steps.

[0150] 4 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 is equipped with a control unit 11, which is a control unit including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, all connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0151] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0152] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a knowledge distillation process. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0153] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via a wired or wireless connection.

[0154] The external device is, for example, a device that transmits a signal used in the knowledge distillation process and that is to be converted by the student model. In such a case, the interface unit 12 acquires the signal used in the knowledge distillation process and that is to be converted by the student model by communicating with the device that transmits the signal used in the knowledge distillation process and that is to be converted by the student model.

[0155] The external device is, for example, the conversion device 2. In such a case, the conversion device 2 can execute the knowledge-distilled student model obtained by executing the knowledge distillation process through communication via the interface unit 12.

[0156] The external device is, for example, a device that stores a teacher model. In such a case, the learning device 1 can execute the teacher model in the knowledge distillation process by communicating with the device that stores the teacher model via the interface unit 12.

[0157] The teacher model may be obtained by learning performed by the control unit 11. In this case, the control unit 11 uses the teacher model obtained by learning performed by the control unit 11 itself in the knowledge distillation process to perform knowledge distillation from the teacher model to a student model.

[0158] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the input devices of the interface unit 12 accept input of various information or signals to the learning device 1 via wired or wireless connections. Note that the information or signals do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0159] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output, for example, information input to an input device of the interface unit 12 or information indicated by a signal, as an image or sound.

[0160] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores, for example, various information generated by the operation of the control unit 11. The storage unit 13 may store a teacher model in advance, for example, before knowledge distillation is performed. The storage unit 13 may exist on a cloud, for example.

[0161] An example of the flow of processing executed by the learning device 1 is as described with reference to FIG.

[0162] 5 is a diagram showing an example of the hardware configuration of the conversion device 2 in an embodiment. The conversion device 2 includes a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected via a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0163] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the conversion device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0164] The control unit 21 controls the operation of each functional unit included in the conversion device 2. The control unit 21, for example, executes a conversion process. The control unit 21, for example, acquires information stored in the storage unit 23. Specifically, the process of acquiring information stored in the storage unit 23 is reading.

[0165] The interface unit 22 includes a communication interface for connecting the conversion device 2 to an external device. The interface unit 22 communicates with the external device via a wired or wireless connection. The external device is, for example, a device that is a source of data to be converted by the conversion process executed by the control unit 21.

[0166] The conversion device 2 executes a knowledge-distilled student model, and therefore is a device that executes inference-stage processing in machine learning theory. Therefore, hereinafter, the conversion target of the conversion processing executed by the control unit 21 is referred to as the inference source.

[0167] When the external device is the device that is the source of the inference source, the interface unit 22 acquires the inference source by communicating with the device that is the source of the inference source.

[0168] The external device may be, for example, the learning device 1. In such a case, the conversion device 2 can execute the knowledge-distilled student model obtained by the learning device 1 through communication via the interface unit 22.

[0169] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, touch panel, microphone, etc. The interface unit 22 may be configured as an interface that connects these input devices to the conversion device 2. In this way, the input devices of the interface unit 22 accept input of various information or signals to the conversion device 2 via wired or wireless connections. Note that the various information or signals that can be input to the communication interface of the interface unit 22 do not necessarily have to be input to the communication interface of the interface unit 22, and may instead be input to the input devices of the interface unit 22.

[0170] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the conversion device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 or information indicated by a signal, as an image or sound.

[0171] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the conversion device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0172] 6 is a flowchart showing an example of the flow of processing executed by the conversion device 2 of the embodiment. The control unit 21 of the conversion device 2 acquires an inference source (step S201). Next, the control unit 21 executes a knowledge-distilled student model on the acquired inference source (step S202). That is, the control unit 21 executes a conversion process on the acquired inference source.

[0173] <Experimental Results> An example of the experimental results will be described. <<Experimental Data>> The experimental data used in the experiment will be described. English speech data from 110 speakers was used as the experimental data. To verify the effectiveness of speech conversion from an unknown speaker to another unknown speaker, data from 10 of the 110 speakers was used for evaluation. Data from the remaining 100 speakers was used for training.

[0174] Each speaker's data consisted of approximately 400 sentences of speech data, of which 10 sentences were used for evaluation to verify the effectiveness of speech conversion from unknown speech content to unknown speech content, and the remaining approximately 390 sentences were used for training. Each speech data was sampled at 22.05 kHz. The feature x was an 80-dimensional logarithmic mel spectrogram extracted from a spectrogram obtained by short-time Fourier transform of the speech data with an FFT size of 1024, a shift width of 256, and a window width of 1024.

[0175] <<Model Structure>> The model structure will be explained. The noise estimator of the teacher model was implemented using U-Net consisting of 12 1D convolution layers. This noise estimator is expressed by the following equation (46).

[0176]

[0177] The activation function used was a gated linear unit (GLU), and the weights of the one-dimensional convolutional layers were normalized using weight normalization.

[0178] We will now explain how each input in the noise estimator in Algorithm 1, which is the noise estimator expressed by the following equation (47), is obtained.

[0179]

[0180] t was obtained by embedding using a sinusoidal positional embedding based on trigonometric functions. ｔｇｔis the target speaker's speech data x using a trained speaker extractor. ｔｇｔ It was obtained by extraction from ｓｒｃ is calculated by using a trained phonetic posterior gram (PPG) estimator to estimate the input speech data x ｓｒｃ Specifically, it was obtained by setting the bottleneck feature of the phoneme posterior probability estimator as p.

[0181] The student model had the same network structure as the teacher model, and was initialized using the teacher model.

[0182] <<Comparative Models>> The comparative models, which are the models to be compared, will now be described. Five comparative models were used in the experiment.

[0183] One of the comparison models was the teacher model A. The teacher model A was a teacher model in which the number of despreading steps K in equation (23) was set to 6.

[0184] The other comparative model was the teacher model B. The teacher model B was a teacher model in which the number of despreading steps K in equation (23) was set to 1.

[0185] The other comparative model was Student Model A. Student Model A is a student model in which the number of steps K' in the de-diffusion process in Equation (28) is set to 1, and the loss function (L ｄｉｓｔｉｌｌ (φ)) was used to perform knowledge distillation.

[0186] The other comparative model was Student Model B. Student Model B is a student model in which the number of steps in the de-diffusion process, K', is set to 1 in Equation (28), and the loss function (L' ａｄｖ (φ), L' ａｄｖ (D'), L' ＦＭ (φ)) was used to perform knowledge distillation.

[0187] The other comparative model was Student Model C. Student Model C is a student model in which the number of steps K' in the de-diffusion process in Equation (28) is set to 1, and the loss function (Lｄｉｓｔｉｌｌ (φ)) and the loss function based on adversarial learning (L´ ａｄｖ (φ), L' ａｄｖ (D'), L' ＦＭ (φ)) and knowledge distillation was performed.

[0188] Due to the difference in the number of dediffusion steps, the conversion speeds of teacher model B, student model A, student model B, and student model C are equivalent, but the conversion speed of teacher model A is six times slower. Note that teacher model A and teacher model B are mathematical models that have not undergone knowledge distillation processing, and student model A, student model B, and student model C are mathematical models that have undergone knowledge distillation processing.

[0189] <<Evaluation Indicators>> The evaluation indices used in the experiment will be explained below. Six evaluation indices were used in the experiment.

[0190] One of the evaluation indices was qMOS, a Mean Opinion Score (MOS) obtained from subjective evaluation experiments on voice quality. It is rated on a five-point scale (5: Excellent, 4: Good, 3: Fair, 2: Poor, 1: Bad), with higher values ​​indicating better quality.

[0191] Another evaluation index was the Mean Opinion Score (sMOS), which is the result of subjective evaluation experiments on speaker similarity. It is rated on a four-point scale (4: Same (sure), 3: Same (not sure), 2: Different (not sure), 1: Different (sure)), and the higher the value, the closer the converted speech is to the target speech in terms of speaker similarity.

[0192] Another evaluation index was UTMOS (UTokyo-SaruLab Mean Opinion Score). This is a Mean Opinion Score (MOS) predicted by a MOS predictor that predicts speech quality. Specifically, the MOS predictor was trained by using speech data with MOS assigned as training data and learning the relationship between the speech data and MOS using a deep learning method. The training data was collected for the purpose of evaluating the quality of speech synthesis and speech conversion, and can predict MOS, which relates to the quality of speech synthesis and conversion. MOS training data takes values ​​from 1 to 5, with higher values ​​indicating better quality.

[0193] Another evaluation index was DNSMOS (Deep Noise Suppression Mean Opinion Score). This is a Mean Opinion Score (MOS) predicted by a MOS predictor that predicts speech quality. Specifically, the MOS predictor was trained by using speech data to which MOS was assigned as training data and learning the relationship between the speech data and MOS using a deep learning method. The training data was collected for the purpose of evaluating the quality of speech noise suppression, and can predict MOS related to the quality of speech noise suppression. The MOS training data takes values ​​from 1 to 5, with higher values ​​indicating better quality.

[0194] Another evaluation index was the character error rate (CER), which is the probability that the speech recognizer will fail to recognize the converted speech. A smaller CER indicates that the speech recognizer is able to convert the converted speech into speech that can be correctly recognized.

[0195] Another evaluation index was SVA, which stands for Speaker Verification Accuracy. SVA is the probability that the speaker verifying device determines that the converted speech and the target speech are from the same speaker. A higher SVA value indicates that the converted speech is closer to the target speech in terms of speaker identity.

[0196] 7 is a diagram showing an example of the results of an experiment in an embodiment. Comparing teacher model A and teacher model B, Fig. 7 shows that teacher model B has significantly lower performance in multiple indices (qMOS, UTMOS) than teacher model A. This shows that simply reducing the number of steps can increase speed, but results in a significant deterioration in performance.

[0197] In contrast, Figure 7 shows that student model A, which underwent knowledge distillation based on a loss function based on a diffusion model, is effective in improving UTMOS, and obtains a score equivalent to that of teacher model A.

[0198] On the other hand, student model B, which underwent knowledge distillation based on a loss function based on adversarial learning, is effective in improving qMOS and DNSMOS, and Figure 7 shows that it achieved a score equivalent to that of teacher model A in DNSMOS.

[0199] Figure 7 shows that Student Model C, which combines Student Model A and Student Model B, is able to utilize the strengths of both models, and outperforms Teacher Model A and Teacher Model B in all scores. In particular, Figure 7 shows that Student Model C has the same conversion speed as Teacher Model B, but is six times faster than Teacher Model A.

[0200] In this way, FIG. 7 shows that knowledge distillation from a teacher model to a student model by knowledge distillation processing enables an increase in conversion speed while maintaining speech quality and speaker similarity.

[0201] The conversion system 100 configured in this manner includes the learning device 1 that executes the knowledge distillation process. Therefore, as described in <Effects of the knowledge distillation process>, the conversion speed can be increased.

[0202] Furthermore, the learning device 1 configured in this manner executes knowledge distillation processing, which makes it possible to increase the conversion speed, as described in <Effects of Knowledge Distillation Processing>.

[0203] Furthermore, the conversion device 2 configured in this manner performs conversion using a knowledge-distilled student model obtained by the knowledge distillation process, thereby enabling the conversion speed to be increased.

[0204] (Variations) Although the explanation so far has been given using voice conversion as an example, the knowledge distillation process is not necessarily applicable only to student models that perform voice conversion, but can be applied to any signal. In other words, for any signal, the knowledge distillation process can be applied to a student model that performs conversion on that signal. The signal may be, for example, an acoustic signal, a biological signal, or another signal acquired by any sensor.

[0205] Therefore, the knowledge distillation process is a process of performing knowledge distillation using, as a teacher model, a mathematical model that performs signal transformation using a diffusion model and that has undergone non-parallel learning. That is, the teacher model is a mathematical model that performs signal transformation using a diffusion model and that has undergone non-parallel learning. Note that non-parallel learning is learning that can be performed in learning mapping from a source to a destination without requiring parallel data of the source and the destination.

[0206] In this case, the student model is a mathematical model that performs signal transformation using a diffusion model and has fewer steps in the dediffusion process than the teacher model.

[0207] In the above knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model.

[0208] The conversion process executed by the conversion device 2 is a process of converting the conversion target using the knowledge-distilled student model obtained by the learning device 1.

[0209] <Another Example of Knowledge Distillation> So far, as an example of knowledge distillation, an example using a loss function including a diffusion loss (hereinafter referred to as "first knowledge distillation") has been described. That is, we have described knowledge distillation in which the student model is updated so as to reduce the difference between the result of transforming a signal in training data using a student model and the result of transforming that result using a teacher model. However, this is just one example.

[0210] The knowledge distillation may be, for example, second knowledge distillation. The second knowledge distillation is knowledge distillation that updates a student model so as to reduce the difference between the result of transforming a signal in training data using a student model and the result of transforming the same signal in training data using a teacher model. In this case, optimization is performed so that the input-output correspondence between the student model and the teacher model matches.

[0211] In this way, in knowledge distillation, the student model may be updated to reduce the difference between the result of transforming a signal by the student model and the result of transforming the signal by the teacher model. Thus, for example, in knowledge distillation, the student model may be updated to reduce the difference between the result of transforming a signal by the student model and the result of transforming that signal or its result by the teacher model.

[0212] In the first knowledge distillation, the teacher model is applied again to the result of conversion by the student model. Therefore, the input / output correspondence between the student model and the teacher model does not necessarily need to match. Therefore, the first knowledge distillation is easier to handle the case where a destination signal corresponding to a source signal is not uniquely determined than the second knowledge distillation.

[0213] We will explain the ease of adaptation. When a destination signal corresponding to a source signal is not uniquely determined, there are multiple solutions for the destination signal. In this case, in the second knowledge distillation, the student model must generate a solution that matches the teacher model, and strong constraints are imposed. On the other hand, in the first knowledge distillation, the student model may generate a solution that does not match the teacher model, and strong constraints are not imposed. In this way, ease of adaptation means that the constraints imposed in the first knowledge distillation are weaker than those in the second knowledge distillation.

[0214] <Another Example of Learning Method for Student Model> As described above, in the diffusion model, the number of steps during conversion can be reduced and conversion can be sped up by replacing the step sequence expressed by equation (22) in the dediffusion process during learning with the subsequence expressed by equation (23). However, as described above, this method involves a large trade-off between speed and quality, and significantly reducing the number of steps during conversion results in a significant reduction in quality.

[0215] Possible methods for speeding up the student model include reducing the number of steps in the dediffusion process compared to the teacher model, reducing the number of parameters compared to the teacher model, or replacing the student model with a network that can perform calculations faster than the teacher model. Speed-up using these methods can be achieved not only by the diffusion model, but also by speeding up the auxiliary information extractor. The auxiliary information extractor is, for example, a language extractor or a speaker extractor. The knowledge distillation from the teacher model to the student model, which will be explained below, can be applied to either method.

[0216] Such techniques can be used to increase speed, and one such technique is the technique described in the above embodiment, but other techniques can also be used to increase speed.

[0217] For example, the control unit may perform a modification distillation process. The modification distillation process is knowledge distillation that uses, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, and that further satisfies a first condition and a second condition. The first condition is that the student model is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model.

[0218] The second condition is that the category to which the signal input to this student model (hereinafter referred to as the "input signal") belongs is not necessarily the same as the category to which the result of this student model's conversion of this signal belongs. Note that the input signal is converted by the student model, and is therefore an example of a signal to be converted by the student model.

[0219] Just to be clear, let us explain that they are not necessarily the same. For example, consider a situation where signal A and signal B are input to a student model. When signal A is input to the student model, the student model converts signal A to generate signal A'. When signal B is input to the student model, the student model converts signal B to generate signal B'. In this case, the second condition asserts that it is acceptable for signal A and signal A' to be in the same category, but signal B and signal B' to be in different categories.

[0220] Note that the condition that the category to which the signal input to the student model belongs is different from the category to which the result of transforming this signal by the student model belongs (hereinafter referred to as the “second' condition”) is included in the second condition. Therefore, the transformation distillation process may be knowledge distillation that uses, as a teacher model, a mathematical model that transforms a signal using a diffusion model and that has been trained by non-parallel transformation, and further satisfies the first condition and the second' condition.

[0221] Each category of the second condition may be any category as long as it is a predetermined category to which the signal belongs. The category may be, for example, the speaker to which the signal belongs. In this case, if the second condition is satisfied, for example, if the signal input to the student model is a speech signal of speech uttered by speaker A, the result of conversion of this signal by the student model is a speech signal of speech uttered by speaker B, who is different from speaker A.

[0222] It should be noted that even when the signal is an audio signal, the category does not necessarily have to be a speaker, but may be, for example, emotion, pronunciation (e.g., native English, non-native English, etc.), acoustic space, or vocalization (e.g., whispering, singing, reading, etc.).

[0223] <<Explanation of an Example of Effect of Modified Distillation Process>> The modified distillation process also increases speed. This is because a mathematical model with fewer calculations than the teacher model is used. A mathematical model with fewer calculations than the teacher model is, for example, a mathematical model with fewer steps in the dediffusion process than the teacher model. A mathematical model with fewer calculations than the teacher model may be, for example, a mathematical model with fewer parameters than the teacher model, a mathematical model that is more compressed than the teacher model, a mathematical model that is more quantized than the teacher model, or a mathematical model that has been replaced with a network that can perform calculations faster than the teacher model.

[0224] In the transformation distillation process, the student model may transform the input signal one or more times. For example, if the input signal is a speech signal of speaker A, the input signal may be transformed once to obtain a speech signal of speaker B, and then the speech signal of speaker B may be further transformed to obtain a speech signal of speaker C, which is different from speakers A and B. In this example, two transformations are performed, but three or more transformations may also be performed.

[0225] Furthermore, for example, if the input signal is a voice signal of speaker A, the input signal may be converted once to obtain a voice signal of speaker A, and then this voice signal of speaker A may be converted again to obtain a voice signal of speaker A. In this example, two conversions are performed, but of course, three or more conversions may also be performed in this example.

[0226] Also, for example, if the input signal is a voice signal of speaker A, this input signal may be converted once to obtain a voice signal of speaker B, and this voice signal of speaker B may then be converted again to obtain a voice signal of speaker A. In this example, two conversions are performed, but of course, three or more conversions may also be performed in this example.

[0227] Furthermore, for example, if the input signal is a voice signal of speaker A, the input signal may be converted once to obtain a voice signal of speaker A, and then this voice signal of speaker A may be converted again to obtain a voice signal of speaker B. In this example, two conversions are performed, but of course, three or more conversions may also be performed in this example.

[0228] In this way, the student model may transform the input signal more than once, under the second condition that the category to which the signal input to the student model belongs is not necessarily the same as the category to which the result of the student model transforming this signal belongs.

[0229] In addition, when a student model transforms an input signal once in the modified distillation process, the student model may be updated in the modified distillation process so as to reduce the difference between the result of the student model transforming this input signal once and the result of the teacher model transforming this result. For example, when an input signal A is input to the student model, the student model may be updated so as to reduce the difference between signal B, which is a signal obtained by the student model transforming input signal A only once, and signal B', which is a signal obtained by the teacher model transforming this signal B.

[0230] In the modification distillation process, the student model may be updated to reduce the difference between the result of multiple transformations of the input signal by the student model and the result of the transformation of this result by the teacher model. For example, when an input signal A is input to the student model, and the student model transforms input signal A once to obtain signal B, and then further transforms signal B to obtain signal C, the student model may be updated to reduce the difference between signal C and signal C', which is the signal obtained by transforming signal C by the teacher model.

[0231] In addition, if the category of the signal obtained by the transformation by the teacher model is different from the category of the signal obtained by each transformation by the student model, the student model may be updated so that the difference between the results of multiple transformations of the signal by the student model and the signal obtained by the teacher model becomes greater.

[0232] For example, when an input signal A is input to a student model, and the student model converts input signal A once to obtain signal B, and then converts signal B again to obtain signal C, the student model may be updated to increase the difference between signal C and signal C'', which is the signal obtained by the teacher model converting signal C. Note that increasing the difference may simply mean increasing the difference between the signals, but may also mean, for example, if the proximity between categories is predetermined, making the category to which signal C belongs and the category to which signal C'' belongs more distant from each other.

[0233] For example, when an input signal A is input to a student model, and the student model converts input signal A once to obtain signal B, the student model may be updated so as to increase the difference between signal B and signal B'', which is a signal obtained by the teacher model converting signal B. Note that increasing the difference may simply mean increasing the difference between the signals, but may also mean, for example, if the proximity between categories is predetermined, making the category to which signal B belongs and the category to which signal B'' belongs more distant from each other.

[0234] <<Explanation Using Mathematical Formulas of an Example of the Transformational Distillation Process>> An example of the transformational distillation process will now be explained using mathematical formulas. First, a method for setting initial parameters will be explained. Below, an example of speaker conversion of speech will be explained, in which a speech signal is used as the signal and a speaker is used as the category, but the same can be applied to other signals and categories.

[0235] <<<Method of Setting Initial Parameters>>> The setting of the initial parameters of the student model will be described. This is the same as the content described in <<<Setting Initial Parameters>>>> in the above <Explanation Using Mathematical Formulas for Knowledge Distillation Processing>>>. However, just to be sure, we will explain it again. The learning parameter of the student model is assumed to be φ. The initial parameter φ during learning of the student model may be set independently of the teacher model, or may be set based on the parameter θ of the teacher model. When set independently of the teacher model, it is set, for example, by randomly sampling a value from a predetermined distribution.

[0236] Furthermore, when the parameters are set based on the parameters of the teacher model, for example, some or all of the parameters φ of the student model are set to be the same as some or all of the parameters θ of the teacher model. Alternatively, when reducing the number of parameters of the student model, the student model may be initialized so that the statistics of the student model match the statistics of the teacher model. Note that the statistics may be any statistics, such as the mean or variance.

[0237] This is similar to the explanation in the above <Explanation Using Formulas for Knowledge Distillation Processing>, but when data is generated based on a student model having φ set in this way as a parameter, the data is shown with φ as a subscript.

[0238] <<<<Knowledge Distillation from Teacher Model to Student Model>>> The step sequence of the dediffusion process of the student model is expressed by the above-mentioned equation (28). As described above, the step sequence S' expressed by equation (28) is a subsequence of the step sequence expressed by equation (22) in the dediffusion process of the teacher model. K' is an integer equal to or less than T.

[0239] Knowledge distillation from the teacher model to the student model is performed using a score distillation loss based on a diffusion model. In the example of the above <Explanation Using Formulas for Knowledge Distillation Process>, the loss function when the student model performs "reconstruction" (hereinafter referred to as "reconstruction score distillation loss") was used as the loss function (or loss) based on the diffusion model. However, the loss function when the student model performs "transformation" (hereinafter referred to as "transformation score distillation loss") may also be used. The reconstruction score distillation loss is specifically the loss expressed by equation (29).

[0240] <<<<<Reconstruction Score Distillation Loss>>>> In the following, in order to explain the transformation score distillation loss, we will first explain the reconstruction score distillation loss. As described above, the reconstruction score distillation loss is expressed by equation (29). In the following explanation, for ease of understanding the transformation score distillation loss, we will use the data x φ and loss L ｄｉｓｔｉｌｌ The reconstruction score distillation loss is represented by the superscript “rec” attached to each of the above. Specifically, in the following description, the reconstruction score distillation loss is represented by the following equation (48).

[0241]

[0242] Here x φ ｒｅｃ is the data generated by the student model (student results). φ ｒｅｃ Specifically, μ expressed by equation (18) θ μ, defined similarly to φAs mentioned above, the superscript src is used when the data is related to the input speech, and the superscript tgt is used when the data is related to the target speech. As mentioned above, the feature of the input speech is expressed as x ｓｒｃ , the speaker information of the target voice is s ｔｇｔ , the linguistic information of the input speech is p ｓｒｃ It can be inferred from this that s ｓｒｃ represents the speaker information of the input speech.

[0243]

[0244] Here, τ satisfies the relationship of the above-mentioned equation (31). Also, the symbol $\epsilon_{\phi}$ in the above-mentioned equation (32) represents the noise estimator of the student model. x τ ｓｒｃ is expressed as x in equation (8). ｔ x ｔ ｓｒｃ Replace with x ０ x ０ ｓｒｃ By replacing the formula with x ０ ｓｒｃ represents the data that has been diffused by τ steps. That is, x τ ｓｒｃ is expressed by the following equation (50).

[0245]

[0246] Also, in formula (48), x θ ｒｅｃ is obtained by the following equation (51).

[0247]

[0248] Here, sg represents a process that does not propagate the gradient, and t satisfies the relationship of the following equation (52): φ,ｔ ｒｅｃ is expressed as x in equation (8). ｔ x φ,ｔ ｒｅｃ Replace with x ０ x φ ｒｅｃ By replacing the formula with x φ ｒｅｃrepresents the data that has been diffused for t steps. That is, x φ,ｔ ｒｅｃ is expressed by the following equation (53).

[0249]

[0250]

[0251] In addition, in equation (48), c(t) represents a term that performs weighting according to t, and may be, for example, a function that decreases in value as t increases. For example, c(t) is expressed as c(t)=α ｔ , or may be expressed by the following equation (54):

[0252]

[0253] Although Equation (48) shows the case where the L2 loss is used, a loss function based on any distance criterion, such as the L1 loss, the Lp loss, the Huber loss, the hinge loss, the cross-entropy loss, the Wasserstein loss, the SSIM (Structural Similarity Index Measure) loss, or the perceptual loss, may be used.

[0254] <<<<<Transformation Score Distillation Loss>>>> The transformation score distillation loss based on the diffusion model is expressed by the following equation (55).

[0255]

[0256] Here, x φ ｃｖ is the data generated by the student model during the transformation process. Specifically, it is expressed by the following equation (56).

[0257]

[0258] The difference from equation (49) is that in equation (49), μ φ As an argument of ｓｒｃ was used, but in equation (56), μ φ As an argument of ｔｇｔ is used.

[0259] Also, in formula (55), x θｃｖ is obtained by the following equation (57).

[0260]

[0261] Here x φ，ｔ ｃｖ is expressed as x in equation (8). ｔ x φ，ｔ ｃｖ Replace with x ０ x φ ｃｖ By replacing the formula with x φ ｃｖ represents the data that has been diffused for t steps. That is, x φ，ｔ ｃｖ is expressed by the following equation (58).

[0262]

[0263] The difference from equation (51) is that in equation (51), μ θ As an argument of ｓｒｃ was used, but in equation (56), μ θ As an argument of ｔｇｔ is used.

[0264] In this way, with conversion score distillation loss, knowledge distillation is performed during the conversion process, making it possible to prevent degradation in speech conversion performance that occurs when input data is output as is. Specifically, if training of a language extractor is performed simultaneously, and the input data is output as is for training, an identity transformation is learned through the language extractor, resulting in degradation in speech conversion performance. In contrast, with conversion score distillation loss, knowledge distillation is performed during the conversion process, so an identity transformation is not learned, making it possible to prevent degradation in speech conversion performance.

[0265] Although Equation (55) shows the case where the L2 loss is used, a loss function based on any distance criterion, such as the L1 loss, the Lp loss, the Huber loss, the hinge loss, the cross-entropy loss, the Wasserstein loss, the SSIM (Structural Similarity Index Measure) loss, or the perceptual loss, may be used.

[0266] In the above description of the reconstruction score distillation loss and the conversion score distillation loss, the values ​​represented by the symbols in τ, t, and equation (9) that appear in equation (50) and the values ​​represented by the symbols in the following equation (59) that appear in equations (53) and (58) have been described as being the same as those used when calculating the reconstruction score distillation loss. However, some or all of these may be sampled independently when calculating the reconstruction score distillation loss and the conversion score distillation loss.

[0267]

[0268] <<<<<Reconversion Score Distillation Loss>>>> Speech conversion is a technology that converts only non-linguistic and paralinguistic information while preserving the linguistic information of the input speech, and it is required that the linguistic information be maintained before and after conversion. One candidate index that shows the performance of preserving this linguistic information before and after conversion is the index that shows how well linguistic information is maintained even when conversion is repeated.

[0269] To improve the performance of speech conversion evaluated by this index, the loss function shown in the following equation (60) (hereinafter referred to as "reconversion score distillation loss") may be used. In other words, by using the reconversion score distillation loss, speech conversion can be performed in which changes in linguistic information due to repeated conversion are suppressed.

[0270]

[0271] Here x φ ｃｖ２ is the data generated by the student model after two transformations. Specifically, it is obtained by the following equation (61).

[0272]

[0273] Here x φ,τ ｃｖ is expressed as x in equation (8). ｔ x φ，ｔ ｃｖ Replace with x ０ x φ ｃｖ By replacing the formula with x φ ｃｖrepresents the data that has been diffused by τ steps. That is, x φ，τ ｃｖ is expressed by the following equation (62).

[0274]

[0275] s ｔｇｔ２ is the speaker information of the target voice in the second conversion.

[0276] In addition, in formula (60), x θ ｃｖ２ is obtained by the following equation (63).

[0277]

[0278] Here x φ,ｔ ｃｖ２ is expressed as x in equation (8). ｔ x φ，ｔ ｃｖ２ Replace with x ０ x φ ｃｖ２ By replacing the formula with x φ ｃｖ２ represents the data that has been diffused for t steps. That is, x φ,ｔ ｃｖ２ is expressed by the following equation (64).

[0279]

[0280] In this way, the retransform score loss expressed by equation (60) has the effect of encouraging the output data obtained by the student model performing two transformations to match the output data of the teacher model. In other words, the retransform score loss expressed by equation (60) can strengthen the retention of linguistic information during transformation.

[0281] Although equation (60) shows the case where the L2 loss is used, any loss function based on a distance criterion, such as the L1 loss, the Lp loss, the Huber loss, the hinge loss, the cross-entropy loss, the Wasserstein loss, the SSIM (Structural Similarity Index Measure) loss, or the perceptual loss, may be used.

[0282] In the above description of the retransformation score distillation loss, the same values ​​are used in the first and second transforms for τ, t, the values ​​represented by the symbols in Equation (9) appearing in Equation (62) etc., and the values ​​represented by the symbols in Equation (59) appearing in Equation (64) etc. However, some or all of these may be sampled independently in the first and second transforms.

[0283] In addition, although the above example shows the application of the retransformation score distillation loss to data generated by a student model after two transformations, the score distillation loss may also be applied to data generated by a student model after three or more transformations.

[0284] <<<<<Inverse Score Distillation Loss>>>> In the above distillation loss, the same speaker information is used in the student model and the teacher model, and this has the effect of encouraging the student model to output data of the same speaker as the teacher model. In order to comparatively strengthen this speaker identity, a loss function expressed by the following equation (65) (hereinafter referred to as "inverse score distillation loss") may be used. In other words, by using the inverse score distillation loss, speaker identity can be comparatively strengthened.

[0285]

[0286] Here, x θ ｃｖ－ｉｎｖ is obtained by the following equation (66).

[0287]

[0288] Here, s ｃｖ－ｉｎｖ represents speaker information extracted from speech uttered by a speaker other than the speaker who uttered the target speech.

[0289] The inverse score distillation loss has the effect of amplifying the difference between the results of a student model converted using certain speaker information and the results of a teacher model converted using other speaker information. Therefore, by using the inverse score distillation loss, it is possible to strengthen the speaker identity of the speech converted by the student model.

[0290] In the above, the speech feature x obtained by one transformation of the student model is φ ｃｖHowever, the speech features obtained by transforming the student model multiple times (for example, if the model is transformed twice, x φ ｃｖ２ ) and apply the inverse score distillation loss. φ ｃｖ２ The inverse score distillation loss for L ｄｉｓｔｉｌｌ ｃｖ２－ｉｎｖ It is expressed as:

[0291] In addition, the speech feature x obtained by reconstructing the student model φ ｒｅｃ Inverse score distillation loss may be applied to x φ ｒｅｃ The inverse score distillation loss for L ｄｉｓｔｉｌｌ ｒｅｃ－ｉｎｖ It is expressed as:

[0292] <<<<<Adversarial Loss>>>> In addition to the distillation loss described above, the adversarial loss expressed by the following equations (67) and (68) may be used.

[0293]

[0294]

[0295] where D represents the classifier, and the real data x ０ and generated data x φ ｒｅｃ On the other hand, the student model minimizes the equation (68) to identify data x that the classifier D cannot distinguish from real data. φ ｒｅｃ In this way, by optimizing the classifier and the student model under competitive conditions, the student model can generate realistic data that cannot be distinguished even by the strengthened classifier.

[0296] Although Equation (67) and Equation (68) show an example of performing adversarial learning using L2 loss, any loss function based on a distance criterion, such as L1 loss, Lp loss, Huber loss, hinge loss, cross-entropy loss, Wasserstein loss, SSIM (Structural Similarity Index Measure) loss, or perceptual loss, may be used.

[0297] Furthermore, as preprocessing for applying the classifier D, a process of converting features into speech using a waveform generator such as a neural vocoder may be executed. Specifically, if the waveform generator is represented by V and the classifier applied after applying the waveform generator is represented by $\mathcal{\hat{D}}$, then equations (67) and (68) can be replaced by the following equations (69) and (70), respectively. Note that $\mathcal{\hat{D}}$ refers to the symbol in the following equation (71). V refers to the symbol in the following equation (94).

[0298]

[0299]

[0300]

[0301] In equation (69), the parameters of the waveform generator V may be fixed, or may be optimized simultaneously when optimizing the student model in equation (70). When optimizing the student model, equations (69) and (70) may be used simultaneously with equations (67) and (68).

[0302] Furthermore, when performing optimization using equations (67) and (68), a loss function expressed by the following equation (72) (hereinafter referred to as a “feature matching loss function”) may also be used at the same time.

[0303]

[0304] where L is the number of layers in the network of the classifier D, and N ｌ is the number of features in the lth layer, D ｌ represents the feature of the lth layer. With this loss function, in the feature space of the classifier D, x ０ and xφ ｒｅｃ It can bring the two closer together.

[0305] Although equation (72) shows the case where L1 loss is used, a loss function based on any distance criterion, such as L2 loss, Lp loss, Huber loss, hinge loss, cross-entropy loss, Wasserstein loss, or SSIM (Structural Similarity Index Measure) loss, may be used.

[0306] Similarly, when performing optimization using equations (69) and (70), the feature matching loss function expressed by the following equation may also be used at the same time.

[0307]

[0308] Here, $\mathcal{\hat{L}}$ represents the number of network layers of the classifier $\mathcal{\hat{D}}$, $\mathcal{\hat{N}_l}$ represents the number of features in the l-th layer, and $\mathcal{\hat{D}_l}$ represents the feature in the l-th layer. Note that $\mathcal{\hat{L}}$ represents the symbol in the following equation (74), $\mathcal{\hat{N}_l}$ represents the symbol in the following equation (75), and $\mathcal{\hat{D}_l}$ represents the symbol in the following equation (76).

[0309]

[0310]

[0311]

[0312] By the loss function of Equation (73), in the feature space of the classifier $\mathcal{\hat{D}}$, x ０ and x φ ｒｅｃ It can bring the two closer together.

[0313] Although equation (73) shows the case where L1 loss is used, a loss function based on any distance criterion, such as L2 loss, Lp loss, Huber loss, hinge loss, cross-entropy loss, Wasserstein loss, or SSIM (Structural Similarity Index Measure) loss, may be used.

[0314] In the above, the data x reconstructed by the student model φ ｒｅｃ We have described the case where an adversarial loss is applied to the data x φ ｃｖ , or a similar adversarial loss may be applied to data that has been transformed multiple times. Note that data that has been transformed multiple times may be, for example, x φ ｃｖ２ In the following, x φ ｃｖ The adversarial loss for x is expressed by the following equation (77), (78), (79), or (80). φ ｃｖ２ The adversarial loss for is expressed by the following formula (81), (82), (83), or (84). When multiple adversarial losses are used, a different classifier may be used for each adversarial loss, or a common classifier may be used.

[0315]

[0316]

[0317]

[0318]

[0319]

[0320]

[0321]

[0322]

[0323] <<<<<Overall Loss Function>>>> The overall loss function is expressed, for example, by the following equations (85) to (87).

[0324]

[0325]

[0326]

[0327] The student model, classifier D, and classifier $\mathcal{\hat{D}}$ are optimized by minimizing Equation (26), Equation (27), and Equation (28), respectively, where the parameters (e.g., λ) before each loss are ｄｉｓｔｉｌｌ ｒｅｃ ) are weighting parameters for each loss that take values ​​greater than or equal to 0, and when they are 0, the associated loss function is ignored. That is, it is not necessary to use all losses in each of Equations (85) to (87). The student model may be trained using only some of the losses.

[0328] Furthermore, in equations (85) to (87), the distillation loss and the adversarial loss are used for data that has been reconstructed, transformed, or transformed twice by the student model. However, the distillation loss or the adversarial loss may be used simultaneously or instead for data that has been transformed three or more times by the student model.

[0329] <<<<<Transformation Using Student Model>>>>> Transformation using the student model is similar to the transformation using the teacher model described above (for example, the transformation according to Algorithm 1 described above).

[0330] Although the application to voice conversion has been described as an example in <Another example of a student model learning method>, it does not necessarily have to be applied to voice conversion. The technology described in <Another example of a student model learning method> is not limited to voice signals and can be applied to any signal, such as acoustic signals, biological signals, or other signals acquired by any sensor.

[0331] If the signal is an acoustic signal, the category may be, for example, an acoustic environment (e.g., noisy environment, anechoic environment, reverberant environment, etc.), a sound effect (e.g., footsteps, door slamming, noise, comfortable sound, etc.), a musical instrument (e.g., piano, violin, etc.), a performance style (e.g., classical, jazz, etc.), or an audio format (e.g., compressed format, mono, stereo, etc.).If the signal is a biological signal, the category may be, for example, subject information, activity information, health condition, stress information, environmental information, or emotion.

[0332] In the section "Another Learning Method for Student Models," an example was described in which a diffusion model was used as the teacher model. However, it is not necessarily a diffusion model, and instead of the diffusion model, a flow matching model or a mean flow model may be used.

[0333] <Experiment in Modified Example> An experiment verifying the effectiveness of the learning method described above in <Another Example of Learning Method for Student Model> will be described.

[0334] The experimental data used was English speech data from 110 speakers. To verify the effectiveness of speech conversion from unknown speakers to unknown speakers, data from 10 of the 110 speakers was used for evaluation, and data from the remaining 100 speakers was used for training.

[0335] In addition, the data for each speaker consisted of approximately 400 sentences of speech data, but in order to verify the effectiveness of speech conversion from unknown speech content to unknown speech content, 10 sentences were used for evaluation and the remaining approximately 390 sentences were used for training.

[0336] Each piece of audio data was sampled at 22.05 kHz. The feature quantity x of each piece of audio data was an 80-dimensional logarithmic mel spectrogram obtained by converting a spectrogram obtained by short-time Fourier transform of the audio data with an FFT size of 1024, a shift width of 256, and a window width of 1024 into a mel scale.

[0337] <<Model Structure>> Using a U-Net consisting of 12 1D convolution layers, the noise estimator $\epsilon_{\theta}$ represented by the symbol in equation (19) was implemented as a training model. A gated linear unit (GLU) was used as the activation function. $\epsilon_{\theta}$ refers to the symbol in equation (46).

[0338] Weight normalization was also used for normalization. Among the inputs of the noise estimator $\epsilon_{\theta}$, t, s, and p were obtained as follows: t was obtained by embedding using sinusoidal positional embedding based on trigonometric functions. s was obtained by extracting from the speech data x using a trained speaker extractor. p was obtained by extracting from the speech data x using a trained phonetic posteriorgram (PPG) estimator. Specifically, the bottleneck feature of the phonetic posterior probability estimator was set to p.

[0339] The student model used a model with the same network structure as the teacher model, except for the network that extracts p. For the network that extracts p, a network p with low computational cost was used to speed up the process. φ Specifically, the teacher model used a conformer-based model, while the student model used a three-layer convolutional neural network.

[0340] The activation function used was the Gated Linear Unit (GLU), and weight normalization was performed using weight normalization and instance normalization. φ was optimized simultaneously with the noise estimator $\epsilon_{\phi}$ of the student model. As mentioned above, $\epsilon_{\phi}$ refers to the symbol in equation (32). The noise estimator $\epsilon_{\phi}$ of the student model was initialized using the noise estimator $\epsilon_{\theta}$ of the teacher model.

[0341] <<Comparison Models>> In the experiment, four models were compared: the baseline, model A', model B', and model C'. In the baseline, the reconstruction score distillation loss (L ｄｉｓｔｉｌｌ ｒｅｃ ) and the adversarial loss. In Model A', the reconstruction score distillation loss (L ｄｉｓｔｉｌｌ ｒｅｃ ) is converted into a score distillation loss (L ｄｉｓｔｉｌｌ ｃｖ ) and training was performed. In Model B', the reconversion score distillation loss (L ｄｉｓｔｉｌｌ ｃｖ２ ) was added to model B and training was performed. ｄｉｓｔｉｌｌ ｃｖ－ｉｎｖ and L ｄｉｓｔｉｌｌ ｃｖ２－ｉｎｖ Training was performed by adding two inverse score distillation losses:

[0342] More specifically, in model C', the losses in the following equations (88) and (89) were used as examples of the losses shown in equations (85) to (87).

[0343]

[0344]

[0345] The value of the weight parameter was the value expressed by the following equation (90).

[0346]

[0347] The loss in model B' is given by L from equations (88) and (89). ｄｉｓｔｉｌｌ ｃｖ－ｉｎｖ and L ｄｉｓｔｉｌｌ ｃｖ２－ｉｎｖ The two reverse scores were those excluding distillation losses.

[0348] Model A' is further derived from Model B' by the retransformed score distillation loss (L ｄｉｓｔｉｌｌ ｃｖ２ ) was excluded.

[0349] The baseline is the transformation score distillation loss (L ｄｉｓｔｉｌｌ ｃｖ ) is the reconstruction score distillation loss (L ｄｉｓｔｉｌｌ ｒｅｃ ) Along with this change, the adversarial loss has also been replaced from the loss in the following equation (91) to the loss in the following equation (92).

[0350]

[0351]

[0352] The number of despreading steps was always 1.

[0353] <<Evaluation Indicators>> Three indices were used in the experiment: dMOS-A, dMOS-B, and SECS. dMOS-A is a Mean Opinion Score (MOS) estimated using a deep learning method to estimate the speech quality of synthesized speech. The higher the value on a five-point scale, the better the quality.

[0354] dMOS-B is the Mean Opinion Score (MOS) estimated by deep learning method B, which estimates the speech quality of noise-removed speech. The higher the value on a five-point scale, the better the quality.

[0355] SECS stands for Speaker Embedding Cosine Similarity, and represents the cosine similarity of the speaker embedding between the converted speech and the target speech. The larger the value, the closer the converted speech is to the target speech in terms of speaker characteristics.

[0356] <<Experimental Results for Modification>> Fig. 8 is a diagram showing an example of the results of an experiment for the modification. When comparing the baseline and invention model A', Fig. 8 shows that invention model A' outperforms the baseline in all scores. Therefore, Fig. 8 shows that conversion score distillation is effective in improving the quality and speaker characteristics of student models.

[0357] Comparing the invention model A' and the invention model B', Figure 8 shows that the SECS remains the same, while dMOS-A and dMOS-B are improved. This shows that the retransformation score distillation is effective in retaining the information that we want to retain even after transformation.

[0358] Comparing the invention model B' and the invention model C', Fig. 8 shows that dMOS-A remains the same, while dMOS-B and SECS improve. This result, especially for the improvement of SECS, indicates that inverse score distillation can improve speaker identity by contrastively emphasizing speaker identity.

[0359] <Regarding Knowledge Distillation Processing and Modification Distillation Processing> As has been explained so far, the control unit 11 may perform knowledge distillation processing or modification distillation processing. As can be seen from the explanation so far, both the knowledge distillation processing and the modification distillation processing are examples of higher-level distillation processing.

[0360] The upper level distillation process is knowledge distillation that uses, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, and that satisfies a higher level condition: the student model in this knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model.

[0361] 9 is a flowchart showing an example of the flow of processing executed by the control unit 11 in the modified example. More specifically, it is a flowchart showing an example of the flow of processing executed in the higher-level distillation process in the modified example. The control unit 11 acquires an input signal (step S301). Next, the control unit 11 performs voice conversion on the input signal using a student model (step S302).

[0362] Next, the control unit 11 performs voice conversion on the result of the voice conversion performed in step S302 using the teacher model (step S303). The control unit 11 updates the student model according to a predetermined rule (step S304). After step S304, the control unit 11 determines whether a predetermined condition for terminating the upper distillation process (hereinafter referred to as the "upper distillation process termination condition") is satisfied (step S305).

[0363] The condition for terminating the upper distillation process may be any condition related to the termination of the upper distillation process, for example, a condition that the loss satisfies a specified condition, a condition that the student model has been updated a specified number of times, a condition that the change in the student model due to the update is smaller than a specified change, or a condition that the data generated by the student model satisfies a specified condition.

[0364] If the upper distillation process termination condition is satisfied (step S305: YES), the upper distillation process is terminated. On the other hand, if the upper distillation process termination condition is not satisfied (step S305: NO), the process returns to step S301.

[0365] <Explanation of an example of the effect of higher-order distillation processing> Higher-order distillation processing also increases speed. This is because a mathematical model with fewer calculations than the teacher model is used. A mathematical model with fewer calculations than the teacher model is, for example, a mathematical model with fewer steps in the dediffusion process than the teacher model. A mathematical model with fewer calculations than the teacher model may be, for example, a mathematical model with fewer parameters than the teacher model, a mathematical model that is more compressed than the teacher model, a mathematical model that is more quantized than the teacher model, or a mathematical model that has been replaced with a network that can perform calculations faster than the teacher model.

[0366] As described above, the control unit 21 performs the conversion process. The conversion process is a process of performing speech conversion using a student model that has undergone knowledge distillation and that has been obtained by the learning device 1. Here, the knowledge distillation into the student model may be a knowledge distillation process. However, as long as it is knowledge distillation, it does not necessarily have to be a knowledge distillation process, and may be an upper distillation process. Therefore, it may also be a transformation distillation process.

[0367] Therefore, the control unit 21 may perform the conversion using, for example, a knowledge-distilled student model obtained by the higher-level distillation process performed by the learning device 1.

[0368] 10 is a flowchart showing an example of the flow of processing executed by the conversion device 2 in the modified example. The control unit 21 of the conversion device 2 acquires an inference source (step S401). Next, the control unit 21 executes a student model that has been knowledge-distilled by a higher-level distillation process on the acquired inference source (step S402). That is, the control unit 21 executes a conversion process on the acquired inference source using a student model that has been knowledge-distilled by a higher-level distillation process.

[0369] <Generation System Also Including a Microphone> Figure 11 is an explanatory diagram illustrating a modified generation system 100a. The generation system 100a may include a learning device 1, a conversion device 2, a microphone 3, and an output device 4. In the generation system 100a, the conversion device 2 and the microphone 3 are communicatively connected, and sound data of sounds picked up by the microphone 3 is transmitted to the conversion device 2. The output device 4 outputs the specified time series generated by the conversion device 2. The output device 4 may be a display device or an audio output device.

[0370] <Other Application Examples> The conversion device 2 may be a device that includes an audio output device such as a speaker, earphones, or headphones, and that outputs inputted voice with changed voice quality.

[0371] The conversion device 2 may be a device that generates audio data for dubbing videos such as movies, animations, etc. For example, the control unit 21 may generate audio data of an audio signal that has been converted from an input audio signal into the voice quality of a character to be dubbed.

[0372] Such a conversion device 2 can reduce the burden of creating dubbing for videos such as movies and animations.

[0373] The conversion device 2 may be a device that converts voice in a call system such as an online conference. More specifically, the conversion device 2 may acquire a voice signal uttered by one speaker of an online conference, convert the voice quality of the acquired signal to generate a voice signal, and transmit this voice signal to the other speaker of the online conference.

[0374] Examples of speech conversion include suppressing noise in the input speech, clarifying the input speech, adding intonation to the input speech, changing the emotion of the input speech, or changing the speaker's personality, etc. This allows the non-verbal information in the input speech to be converted to what the user intended, anonymized, or to play a virtual character.

[0375] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0376] Note that all or part of the functions of the learning device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media (ROM, RAM, etc.) such as magnetic disks, magneto-optical disks, optical disks (CD-ROM, DVD-ROM, etc.), semiconductor memories (volatile memory, non-volatile memory, etc.), and storage devices such as hard disks built into computer systems. The program may also be transmitted via telecommunications lines.

[0377] The control unit 21 is an example of a conversion unit.

[0378] Note that c(t) in equation (29) may be expressed by the following equation (93).

[0379]

[0380]

[0381] The condition that the number of steps in the dediffusion process is smaller than that of the teacher model is included in the condition that the number of calculations is smaller than that of the teacher model. This is as explained in "Example of Effects of Higher-Level Distillation Processing." Therefore, the first learning device is a type of second learning device.

[0382] The first learning device includes a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned through non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model, and wherein, in the knowledge distillation, the student model is updated so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

[0383] The second learning device includes a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned through non-parallel transformation, wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model, and the student model is updated in the knowledge distillation so as to reduce the difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

[0384] The following additional notes are provided regarding the above-described embodiments.

[0385] (Supplementary Item 1) A learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model.

[0386] (Supplementary Item 2) The learning device according to Supplementary Item 1, wherein a category to which a signal input to the student model belongs is not necessarily the same as a category to which a result of the student model converting the signal belongs.

[0387] (Supplementary Item 3) The learning device according to Supplementary Item 1, wherein the student model transforms a signal input to the student model one or more times.

[0388] (Supplementary Item 4) The learning device according to Supplementary Item 2, wherein the student model is updated so as to reduce the difference between a result of the student model transforming a signal input to the student model once and a result of the teacher model transforming the result.

[0389] (Supplementary Item 5) The learning device according to Supplementary Item 3, wherein in the knowledge distillation, the student model is updated so as to reduce a difference between a result of multiple transformations of a signal input to the student model by the student model and a result of the transformation of the result by the teacher model.

[0390] (Supplementary Item 6) The learning device according to Supplementary Item 1, wherein in the knowledge distillation, the student model is updated so as to reduce a difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

[0391] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0392] REFERENCE SIGNS LIST 100, 110a...conversion system, 1...learning device, 2...conversion device, 11...control unit, 12...interface unit, 13...storage unit, 21...control unit, 22...interface unit, 23...storage unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. A learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation; wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in a de-diffusion process than the teacher model; and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between a result of signal transformation by the student model and a result of signal transformation by the teacher model.

2. A conversion device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation, wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model, and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model.

3. A learning method executed by a learning device, comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been trained by non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer steps in the de-diffusion process than the teacher model, and wherein, in the knowledge distillation, the student model is updated so as to reduce the difference between the result of signal transformation by the student model and the result of signal transformation by the teacher model, the learning method comprising: a knowledge distillation step in which the control unit performs the knowledge distillation.

4. A program for causing a computer to function as the learning device according to claim 1.

5. A learning device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of calculations than the teacher model.

6. The learning device according to claim 5, wherein the category to which the signal input to the student model belongs is not necessarily the same as the category to which the result of transforming the signal by the student model belongs.

7. The learning device of claim 5, wherein, when the category of the signal obtained by the transformation by the teacher model is different from the category of the signal obtained by each of the transformations by the student model, the knowledge distillation updates the student model so that the difference between the result of multiple transformations of the signal by the student model and the signal obtained by the teacher model becomes larger.

8. A conversion device comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation; wherein the student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has a smaller number of operations than the teacher model; and a conversion unit that performs conversion using a knowledge-distilled student model obtained by a learning device.

9. A learning method executed by a learning device, comprising: a control unit that performs knowledge distillation using, as a teacher model, a mathematical model that transforms signals using a diffusion model and that has been learned by non-parallel transformation, wherein a student model in the knowledge distillation is a mathematical model that transforms signals using a diffusion model and that has fewer operations than the teacher model, the learning method comprising: a knowledge distillation step in which the control unit performs the knowledge distillation.

10. A program for causing a computer to function as the learning device according to any one of claims 5 to 7.

Images