Learning device, inference device, learning method, and program

Abstract

Provided is a learning device comprising a control unit that performs training of a learning target, wherein: in the training, training is performed using a training model, which is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel-spectrogram extractor; and the learning target is a mathematical model that is a part of the training model and differs from that obtained as a mathematical model used in an inference stage by the PRVAE-VC in that a HuBERT model is used instead of a mel-spectrogram extractor.

Description

Learning device, inference device, learning method and program

[0001] The present invention relates to a learning device, an inference device, a learning method, and a program.

[0002] Speech conversion is a technology that converts only the non-linguistic and paralinguistic components (such as speaker identity and speaking style) of input speech while preserving the linguistic information (spoken sentences) of the input speech. It can be applied to speaker identity conversion, speech assistance, voice enhancement, accent conversion, and more.

[0003] There are two frameworks for training conversion models: parallel speech conversion and non-parallel speech conversion. Parallel speech conversion is a framework that requires speech with the same speech content from the input speaker and the target speaker. Since the target speech exists during training, it has the advantage of being able to directly optimize the conversion function of the acoustic features, but the cost of collecting training data is high. On the other hand, non-parallel speech conversion is a more difficult task than parallel speech conversion because the conversion function cannot be directly optimized due to the absence of the target speech. However, it has the advantage of being able to use a large speech corpus without any restrictions on the speech content, and has been attracting attention in recent years.

[0004] Non-parallel speech conversion consists of an encoder that performs a speech recognition function and a decoder that performs a text-to-speech synthesis function. The encoder extracts intermediate features from the input speech, and the decoder generates features for the converted speech by adding target speaker information to the intermediate features. Non-parallel speech conversion assumes that finding better intermediate features as the encoder's output enables the model to generalize more effectively to unknown data and different speakers.

[0005] For example, in PRVAE-VC (see Non-Patent Document 1), speech that has the same speech content as the input speech but different speaker information (for example, speech with a lower pitch) is generated, and so-called PR regularization is explicitly introduced.

[0006] To be clear, PR regularization is a constraint on perturbation tolerance that brings intermediate features of input speech closer to those of artificially generated speech. As a result of PR regularization, PRVAE-VC can extract features with low speaker dependency (i.e., high perturbation tolerance).

[0007] On the other hand, HuBERT (see Non-Patent Document 2) allows discretization of content and speaker information. More specifically, discretization is a process that creates an information bottleneck that separates content and speaker information. The discretization procedure in HuBERT is explained below. In HuBERT, the model captures the most salient features of the speech signal while discarding redundant and less informative details. As a result, the modeled speech representation is compact and rich in semantic information.

[0008] Both PRVAE-VC and HuBERT compress the semantics of input information, but they differ in terms of the dimension along which the compression is performed.

[0009] K. Tanaka, H. Kameoka, and T. Kaneko, “PRVAE-VC: Non-Parallel Many-to-Many Voice Conversion with Perturbation-Resistant Variational Autoencoder,” in ISCA Speech Synthesis Workshop, 2023.W.-N. Hsu, B. Bolte, Y.-HH Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 6 3451-3460, 2021.

[0010] However, there is a growing demand for more generalized intermediate features, which are highly tolerant to perturbations and discretize content and speaker information.

[0011] In view of the above circumstances, an object of the present invention is to provide a technique for obtaining more generalized intermediate features.

[0012] One aspect of the present invention is a learning device that includes a control unit that learns a learning object, and the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that it uses a HuBERT model instead of a mel spectrogram extractor, and the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model to be used in the inference stage in that it uses a HuBERT model instead of a mel spectrogram extractor.

[0013] One aspect of the present invention is an inference device comprising: a control unit that learns a learning object; the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel spectrogram extractor; the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor; and an inference unit that performs inference using the mathematical model obtained by the learning device after the learning by the learning device.

[0014] One aspect of the present invention is a learning method executed by a learning device, the learning method including a control unit that learns a learning object, wherein the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel spectrogram extractor, and the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor, the learning method including a learning step in which the control unit performs the learning.

[0015] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0016] According to the present invention, it is possible to obtain more generalized intermediate features.

[0017] An explanatory diagram for explaining an inference system of an embodiment. A flowchart showing an example of the flow of processing executed by a control unit in an embodiment. A diagram showing algorithm 1 in an embodiment. A diagram showing an example of the hardware configuration of a learning device in an embodiment. A diagram showing an example of the hardware configuration of an inference device in an embodiment. A flowchart showing an example of the flow of processing executed by an inference device of an embodiment. A diagram showing an example of the results of an experiment in an embodiment. A flowchart showing an example of the flow of processing executed by an inference device of an embodiment.

[0018] 1 is an explanatory diagram illustrating an inference system 100 according to an embodiment. The inference system 100 includes a learning device 1 and an inference device 2. The learning device 1 includes a control unit 11 including a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus. The control unit 11 executes a learning process.

[0019] The learning process is a process of learning a mathematical model to be learned (hereinafter referred to as the "learning model"). In the learning, a mathematical model (hereinafter referred to as the "learning model") is learned, which differs from PRVAE-VC in that a HuBERT model is used instead of a mel-spectrogram extractor. Here, the HuBERT model has been trained in advance. The learning model is a mathematical model that is a part of the learning model.

[0020] The training model will be described in more detail. The training model executes HuBERT on the speech signal. The HuBERT model has been trained in advance. The training model performs content encoding and speaker encoding on the results of the HuBERT execution. Then, based on information indicating the content obtained by the content encoding and information indicating the speaker obtained by the speaker encoding, the training model obtains a speech signal of the speaker's speech indicating the content.

[0021] Therefore, the trained learning target model is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor.

[0022] Incidentally, the loss function in the learning process includes a negative log-likelihood loss (hereinafter referred to as "likelihood loss") related to the Gaussian distribution and a PR loss.

[0023] The PR loss is the latent space variable z of the speech signal obtained by the content-preserving transformation in PRVAE-VC. ｍ and the distribution of the latent space variable z of the speech signal before the content-preserving conversion. Just to clarify, content-preserving conversion is a process of converting the speech indicated by a speech signal while preserving the linguistic information indicated by the speech signal to be converted. Therefore, content-preserving conversion can be said to be a process of generating speech that has the same speech content as the input speech but different speaker information. Note that generating speech means generating a speech signal.

[0024] An example of a mathematical expression of the PR loss will be described later. Note that the difference may be, for example, a predetermined divergence, such as the KL (Kullback-Leibler) divergence.

[0025] The training model is a mathematical model that differs from PRVAE-VC in that it uses a HuBERT model instead of a mel-spectrogram extractor. Therefore, to explain the training model more specifically, the training target model is a mathematical model that performs content-preserving conversion, a first HuBERT execution process, a first speaker encoding, a first content encoding, a second HuBERT execution process, a second speaker encoding, a second content encoding, and a first decoding.

[0026] In the training model, the target of content-preserving transformation is the speech signal input to the training model.

[0027] The first HuBERT execution process is a process of executing HuBERT on the result of the content-preserving conversion. Note that this HuBERT model has been trained in advance. HuBERT is a mathematical model that obtains the distribution of feature quantities of a signal input to its own model (i.e., HuBERT). Hereinafter, the feature quantities obtained by HuBERT are referred to as HuBERT feature quantities.

[0028] The first speaker encoding is a speaker encoding process that obtains a speaker vector from the result of the first HuBERT execution process. Note that speaker encoding is an encoding process that obtains a speaker vector from an encoding target that is the target of encoding. The encoding target of speaker encoding is information that indicates an audio signal, such as the audio signal itself or features of the audio signal. A speaker vector is a vector that indicates the speaker of the audio indicated by the encoding target. Therefore, speaker encoding is a process that obtains information that indicates the speaker of the audio indicated by the encoding target. Therefore, the first speaker encoding obtains information that indicates the speaker of the audio indicated by the audio signal obtained by content-preserving conversion. Note that information that indicates the speaker is speaker information.

[0029] The first content encoding is performed based on the results of the first HuBERT execution process and the results of the first speaker encoding, and converts the latent space variables z ｍ This is the content encoding process that obtains the distribution of the latent space variable z ｍ The distribution of, for example, the latent space variable z ｍTherefore, the latent space variable z ｍ For example, obtaining the distribution of the latent space variable z ｍ This is a process to obtain the mean and logarithmic variance of the distribution of the latent space variable z ｍ The distribution of the latent space variable z ｍ Any statistical value may be used as long as it is expressed as a statistical value of the distribution of the above.

[0030] Content encoding is an encoding process that obtains a vector (hereinafter referred to as a "content vector") indicating the content of the audio represented by the encoding target from the encoding target. The encoding target in content encoding is information indicating the audio signal itself or features of the audio signal. Therefore, content encoding is a process that obtains information (hereinafter referred to as "content information") indicating the content of the audio utterance represented by the encoding target.

[0031] The second HuBERT execution process is a process of executing HuBERT on the conversion target (i.e., the target of content-preserving conversion). Note that this HuBERT model has also been trained in advance. Since this conversion target is the target of content-preserving conversion, it is the speech signal before conversion by content-preserving conversion.

[0032] The second speaker encoding is a speaker encoding process that obtains a speaker vector from the result of the second HuBERT execution process, and therefore obtains speaker information indicating the speaker of the speech represented by the speech signal before the content-preserving conversion.

[0033] The second content encoding is a content encoding process that obtains the distribution of the latent space variable z of the speech signal before transformation by the content-preserving transformation based on the result of the second HuBERT execution process and the result of the second speaker encoding.

[0034] The first decoding is a decoding process for obtaining a distribution of HuBERT features of the speech signal before conversion by content-preserving conversion, based on the result of the second speaker encoding and the result of the second content encoding. The distribution of HuBERT features may be expressed, for example, by the mean and logarithmic variance of the distribution of HuBERT features. Therefore, obtaining the distribution of HuBERT features is a process for obtaining, for example, the mean and logarithmic variance of the distribution of HuBERT features. Note that the distribution of HuBERT features may be expressed by any statistical value as long as it is expressed by the statistical value of the distribution of HuBERT features.

[0035] As mentioned above, the training model is part of the training model. More specifically, the training model is a mathematical model that executes the second HuBERT execution process, the second speaker encoding, the second content encoding, and the first decoding. Through learning using the training model, the content of the first speaker encoding, the first content encoding, the second speaker encoding, the second content encoding, and the first decoding processes is updated. When a predetermined condition for terminating such learning (hereinafter referred to as the "learning termination condition") is met, the training model included in the training model is the trained training model.

[0036] Therefore, the trained learning model is a mathematical model that executes the second HuBERT execution process, the second speaker encoding, the second content encoding, and the first decoding when the learning termination condition is met.

[0037] The learning termination condition may be any condition related to the termination of the learning process, such as a condition that the learning model has been updated a predetermined number of times. The learning termination condition may also be a condition that the change in the learning model due to the update is smaller than a predetermined change. "Completed learning" means the point in time when the learning termination condition is met.

[0038] The inference device 2 executes an inference process, which is a process of inferring intermediate features of an inference target speech signal of the intermediate features using a mathematical model obtained by the learning device 1 through a learning process by the learning device 1. More specifically, the inference process is a process of inferring intermediate features of an inference target speech signal of the intermediate features using either a trained learning target model obtained by the learning process or a knowledge-distilled student model obtained by knowledge distillation, which will be described later.

[0039] <Comparison with PRVAE-VC> Here, in order to deepen understanding of the learning model, we will first explain PRVAE-VC, which is a well-known technique, and then describe the differences between PRVAE-VC and the learning model.

[0040] 2 is an explanatory diagram illustrating a learning model in an embodiment. More specifically, FIG. 2 is an explanatory diagram of PRVAE-VC for deepening understanding of the learning model.

[0041] PRVAE-VC is a mathematical model that performs a content-preserving transformation, a first mel spectrogram extraction process, a third speaker encoding, a third content encoding, a second mel spectrogram extraction process, a fourth speaker encoding, a fourth content encoding, and a second decoding.

[0042] The first mel spectrogram extraction process is a process in which a mel spectrogram extractor obtains a mel spectrogram of the audio signal obtained by content-preserving conversion.

[0043] The third speaker encoding is a speaker encoding process that differs from the first speaker encoding in that the target of encoding is not the result of the first HuBERT execution process but the result of the first mel spectrogram extraction process.

[0044] The third content encoding is a content encoding process that differs from the first content encoding in that the encoding target is a pair of the result of the first mel spectrogram extraction process and the result of the third speaker encoding, instead of a pair of the result of the first HuBERT execution process and the result of the first speaker encoding.

[0045] The second mel spectrogram extraction process is a process in which a mel spectrogram of the audio signal before conversion by the content-preserving conversion is obtained by a mel spectrogram extractor.

[0046] The fourth speaker encoding is a speaker encoding process that differs from the second speaker encoding in that the encoding target is not the result of the second HuBERT execution process but the result of the second mel spectrogram extraction process.

[0047] The fourth content encoding is a content encoding process that differs from the second content encoding in that the encoding target is a pair of the result of the second mel spectrogram extraction process and the result of the fourth speaker encoding, instead of a pair of the result of the second HuBERT execution process and the result of the second speaker encoding.

[0048] The second decoding is a decoding process that differs from the first decoding in that the decoding target is a pair of the result of the fourth speaker encoding and the result of the fourth content encoding, instead of a pair of the result of the second speaker encoding and the result of the second content encoding.

[0049] In PRVAE-VC, the mathematical model that performs the second mel spectrogram extraction process, the fourth speaker encoding, the fourth content encoding, and the second decoding is the mathematical model used in the inference stage (hereinafter referred to as the "model used in the inference stage").

[0050] Thus, PRVAE-VC differs from the training model in that it uses a mel-spectrogram extractor instead of the HuBERT model.

[0051] <Explanation Using Mathematical Formulas> The mel spectrogram obtained by the mel spectrogram extractor is denoted as x, the speaker vector is denoted as c, and the speaker encoding, content encoding, and decoding in PRVAE-VC are explained using mathematical formulas.

[0052] Speaker encoding in PRVAE-VC is expressed, for example, by the following equation (1).

[0053]

[0054] Content encoding in PRVAE-VC is expressed, for example, by the following formula (2), and decoding in PRVAE-VC is expressed, for example, by the following formula (3).

[0055]

[0056]

[0057]

[0058] where ε represents Gaussian noise, [;] represents concatenation along the channel dimension, and the symbols in equation (5) below represent element-wise multiplication.

[0059]

[0060] The meaning of the subscripts is explained below. μ ｙ denotes the mean μ for the distribution of y, and σ ｙ means the standard deviation of the distribution of y, and σ ２ ｙ means the variance for the distribution of y.

[0061] A distinctive feature of PRVAE-VC is that it trains the latent variable representations of the original speech and the speech processed by content-preserving conversion so that they match as closely as possible. ０ Mel spectrogram x extracted from processed audio obtained by shifting and frequency warping ｍ As a result, the latent space variables z ｍ The parameters of the distribution of the mel spectrogram x ｍ The speaker encoding for ｍ The content encoding for is expressed, for example, by the following equation (7).

[0062]

[0063]

[0064] Latent space variables z and z ｍIn PRVAE-VC, the PR loss is defined by the following formula (8) so that the content encoding process is learned so as to match the two distributions of (a) and (b). Note that here, a formula using the KL divergence as the divergence is exemplified, but the divergence may be another divergence such as the L1 distance.

[0065]

[0066] where N(·) represents the Gaussian distribution and F ＫＬＤ represents the KL divergence loss function, and λ ｐｒ represents the regularization parameter that controls the importance of PR regularization.

[0067] Therefore, the loss function L in PRVAE-VC is given by the following equations (9) to (11).

[0068]

[0069]

[0070]

[0071] Here, F ＧＮＬＬ represents the loss function of the negative log-likelihood for a Gaussian distribution.

[0072] <Loss Function in Learning of Learning Model> Using the mathematical symbols that have been used so far, an example of a loss function in learning of a learning model can be expressed by the following equation (12), for example.

[0073]

[0074] In this way, the loss function L in learning the learning model ｔｅａｃｈｅｒ Unlike the loss function L in PRVAE-VC, the loss term L ｚ may not include.

[0075] <Effects of the Learning Process> Here, the effects of the learning process will be described. As explained so far, in the learning process, a learning model is learned that differs from PRVAE-VC in that a HuBERT model is used instead of a mel-spectrogram extractor. Therefore, in this learning, PR regularization is explicitly performed, and discretization using HuBERT is also performed. Therefore, this type of learning process can be said to be learning that additively takes into account multiple different semantic compressions, namely perturbation tolerance and discretization. Furthermore, because PR regularization and discretization are performed in the learning process, the trained learning model can obtain more generalized features than PRVAE-VC or HuBERT.

[0076] <Knowledge Distillation> Incidentally, the control unit 11 may further perform knowledge distillation using the learning target model as a teacher model after the learning process is completed (i.e., after the learning completion condition is satisfied). Knowledge distillation may be performed in any manner, but for example, knowledge distillation may be performed using a learning model, as shown in the following FIG.

[0077] 3 is an explanatory diagram illustrating an example of knowledge distillation in an embodiment. In the knowledge distillation in the example of FIG. 3, a trained training model is used. A training object model for training is included in a part of the trained training model, and this training object model for training is a teacher model.

[0078] The student model in knowledge distillation in the example of FIG. 3 performs a third mel-spectrogram extraction process, a fifth content encoding process, a third HuBERT execution process, a fifth speaker encoding process, and a third decoding process.

[0079] The third mel spectrogram extraction process is a process of acquiring a mel spectrogram of a target speech signal by a mel spectrogram extractor. The target speech signal in the third mel spectrogram extraction process is the same as the speech signal input to the teacher model.

[0080] The third HuBERT execution process is a process of executing HuBERT on the speech signal input to the student model.

[0081] The fifth speaker encoding is a speaker encoding process that obtains a speaker vector from the result of the third HuBERT execution process, and therefore obtains speaker information indicating the speaker of the voice represented by the voice signal input to the student model.

[0082] The fifth content encoding is a content encoding process that obtains a content vector from the mel spectrogram obtained by the third mel spectrogram extraction process based on the results of the fifth speaker encoding. Therefore, the fifth content encoding obtains content information that indicates the speech content of the voice represented by the voice signal input to the student model.

[0083] The fifth content encoding and the fifth speaker encoding are in a Stop gradient relationship. The Stop gradient relationship is defined as not passing gradient information to the model parameters on the path connecting the Stop gradient and the input. This relationship prevents the model parameters from being updated during training.

[0084] The third decoding is a decoding process for obtaining a distribution of features of the mel spectrogram of the speech signal input to the student model based on the results of the fifth speaker encoding and the fifth content encoding.

[0085] The third decoding and the fifth speaker encoding are in a stop gradient relationship.

[0086] In the knowledge distillation in the example of FIG. 3, for example, the loss function of the following equation (13) is used as the loss function.

[0087]

[0088]

[0089] N ｔ represents a Gaussian distribution with distribution parameters obtained from the teacher model. ｓ represents a Gaussian distribution with distribution parameters obtained from the student model. ｄｉｓｔｉｌThe effects of discretization by Hubert and regularization by PRVAE-VC are passed on to the student model. Note that the fifth speaker encoder may be the speaker encoder of the teacher model (i.e., the trained second speaker encoder), and in this case, the fifth speaker encoder may be frozen during knowledge distillation.

[0090] Knowledge distillation is performed until a predetermined condition for terminating knowledge distillation (hereinafter referred to as the "knowledge distillation termination condition") is satisfied. The knowledge distillation termination condition may be any condition for terminating knowledge distillation, for example, a condition that the student model has been updated a predetermined number of times. The knowledge distillation termination condition may be, for example, a condition that the change in the student model due to the update is smaller than a predetermined change. The student model at the time when the knowledge distillation termination condition is satisfied is a student model that has undergone knowledge distillation.

[0091] Knowledge distillation produces a student model that is lighter than the teacher model, so the student model produced by knowledge distillation is more suitable for real-time speech conversion with lower latency than the teacher model.

[0092] <Experimental Results> Here, an example of the experimental results will be described. In the experiment, the content encoder that performs content encoding, the speaker encoder that performs speaker encoding, and the decoder that performs decoding were all implemented using two-layer LSTM.

[0093] In the experiment, the following six systems were evaluated. One of the systems evaluated was "Recorded: recorded audio." The system "Recorded: recorded audio" is defined as the raw audio at the time of recording, audio in a state where no processing has been applied. One of the systems evaluated was "AnaSyn: analyzed and resynthesized audio." The system "AnaSyn: analyzed and resynthesized audio" is defined as audio obtained by resynthesizing the waveform of the features analyzed from the above-mentioned raw audio without performing feature conversion. One of the systems evaluated was PRVAE-VC.

[0094] One of the systems to be evaluated is a loss function with L ｚThe model to be trained was shown in Figure 1, which was obtained by a training process that did not include L. This model is the "teacher model" with No. T1 in Figure 4, which will be described later. One of the systems to be evaluated also had L in the loss function. ｚ This model is the training model shown in FIG. 1, which was obtained by a training process including the following. This model is shown in FIG. 4, which will be described later, as the "teacher model w / L" with No. T2. ｚ One of the systems evaluated was the student model shown in FIG.

[0095] In the experiment, 1,000 sentences out of 1,132 sentences from the English speech dataset CMU ARCTIC dataset (see Reference 1) were used as training data and 132 sentences as evaluation data. Three objective evaluation experiments were conducted: evaluation of speech recognition error rate, evaluation of speaker similarity, and evaluation of speech quality.

[0096] Reference 1: J. Kominek and A.W. Black, “The CMU Arctic speech databases,” in ISCA Speech Synthesis Workshop, pp. 223-224, 2004.

[0097] The speech sampling rate was 16 kHz. In the experiment, the Character Error Rate (CER) was calculated as the speech recognition error rate. The lower the error rate, the better the quality.

[0098] For the voice quality assessment, a Mean Opinion Score (MOS) was predicted using a deep learning approach to predict voice quality scores. A higher MOS score indicates better voice quality.

[0099] To evaluate speaker similarity, Speaker Similarity (SIM) was predicted using a deep learning method. A higher SIM value indicates a higher speaker similarity.

[0100] Fig. 4 is a diagram showing an example of experimental results in an embodiment. Comparing the results of system B and system T2 in Fig. 4, the results in Fig. 4 show that T2 is superior to B in all evaluation indices. In this way, Fig. 4 shows that HuBERT and PR regularization are additively effective. Furthermore, the results of T1 and T2 shown in Fig. 4 show that it is better not to include the loss term Lz in the loss function.

[0101] Comparing the results of System B and System S2 in Figure 4, the results in Figure 4 show that the effects of Hubert and PR regularization are carried over to the student model by performing the training process and knowledge distillation after the training process. Because S2 is a model of the same size as B, it is guaranteed that the student model will also operate in the environment in which real-time speech inference system B operates.

[0102] In FIG. 4, the values ​​following ± indicate 95% confidence intervals.

[0103] 5 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 is equipped with a control unit 11, which is a control unit including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, all connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0104] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0105] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a learning process. The control unit 11 may execute, for example, knowledge distillation. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0106] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via a wired or wireless connection.

[0107] The external device may be, for example, a speech signal used in the training process and input to the training model (hereinafter referred to as a "first training speech signal"). In such a case, the interface unit 12 acquires the training speech signal by communicating with the device that is the sender of the first training speech signal.

[0108] The external device may be, for example, a speech signal used in knowledge distillation and input to the teacher model and the student model (hereinafter referred to as the "second training speech signal"). In such a case, the interface unit 12 acquires the second training speech signal by communicating with the device that is the sender of the second training speech signal. Note that the first training speech signal and the second training speech signal may be the same or different.

[0109] The external device is, for example, the inference device 2. In such a case, when the control unit 11 executes the learning process but does not execute knowledge distillation, the inference device 2 can execute the trained learning object model obtained by executing the learning process through communication via the interface unit 12.

[0110] Furthermore, when the control unit 11 performs the learning process and knowledge distillation, the inference device 2 can execute the student model through communication via the interface unit 12. When the control unit 11 performs the learning process and knowledge distillation, the inference device 2 may execute the trained learning target model obtained by executing the learning process through communication via the interface unit 12.

[0111] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the input devices of the interface unit 12 accept input of various information or signals to the learning device 1 via wired or wireless connections. Note that the information or signals do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0112] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output, for example, information input to an input device of the interface unit 12 or information indicated by a signal, as an image or sound.

[0113] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may exist on a cloud, for example.

[0114] 6 is a flowchart showing an example of the flow of processing executed by the learning device 1 according to the embodiment. The example in FIG. 6 shows an example of the flow of processing when knowledge distillation is executed after the execution of the learning processing until the learning termination condition is satisfied.

[0115] The control unit 11 acquires a set of first training speech signals (step S101). Note that the number of elements in the set may be one. Next, the control unit 11 executes a training process using the set of first training speech signals until a training termination condition is met (step S102). Next, the control unit 11 acquires a set of second training speech signals (step S103). Next, the control unit 11 executes knowledge distillation using the set of second training speech signals until a knowledge distillation termination condition is met (step S104).

[0116] 7 is a diagram showing an example of the hardware configuration of the inference device 2 in an embodiment. The inference device 2 is equipped with a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected by a bus, and executes a program. By executing the program, the inference device 2 functions as a device equipped with the control unit 21, an interface unit 22, and a memory unit 23.

[0117] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the inference device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0118] The control unit 21 controls the operation of each functional unit included in the inference device 2. The control unit 21, for example, executes an inference process. The control unit 21, for example, acquires information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.

[0119] The interface unit 22 is configured to include a communication interface for connecting the inference device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits an inference target in the inference process executed by the control unit 21. The inference target is an audio signal. In such a case, the control unit 21 acquires the inference target by communication via the interface unit 22.

[0120] The external device may be, for example, the learning device 1. In such a case, the control unit 21 can execute the learned learning object model obtained by the learning device 1 through communication via the interface unit 22. Note that if the control unit 11 also performs knowledge distillation, the control unit 21 can execute the knowledge-distilled student model obtained by the learning device 1 through communication via the interface unit 22.

[0121] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, touch panel, etc. The interface unit 22 may be configured as an interface that connects these input devices to the inference device 2. In this way, the input devices of the interface unit 22 accept input of various information or signals to the inference device 2 via wired or wireless connections. Note that the various information or signals that can be input to the communication interface of the interface unit 22 do not necessarily have to be input to the communication interface of the interface unit 22, but may instead be input to the input devices of the interface unit 22.

[0122] The interface unit 22 outputs, for example, various types of information. The interface unit 22 is configured to include, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the inference device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 or information indicated by a signal, as an image or sound.

[0123] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the inference device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0124] 8 is a flowchart showing an example of the flow of processing executed by the inference device 2 of the embodiment. The control unit 21 of the inference device 2 acquires an inference target (step S201). Next, the control unit 21 executes inference processing on the acquired inference target (step S202).

[0125] As described above, the inference process is an inference using a mathematical model obtained by the learning device 1 through the learning process by the learning device 1. Therefore, if the control unit 11 has performed up to the knowledge distillation process, the inference process of step S202 may be a process of executing a knowledge-distilled student model on the inference target. Also, if the control unit 11 has performed the learning process but not the knowledge distillation process, the inference process of step S202 may be a process of executing a trained learning target model on the inference target. Note that even if the control unit 11 has performed up to the knowledge distillation process, the inference process may be a process of executing a trained learning target model on the inference target.

[0126] The inference system 100 configured in this way executes a learning process, which makes it possible to obtain more generalized intermediate features, as described in <Effects of the Learning Process>.

[0127] The learning device 1 configured as above also executes a learning process, which makes it possible to obtain more generalized intermediate features, as described in <Effects of the Learning Process>.

[0128] Furthermore, the inference device 2 configured in this way performs inference using the mathematical model obtained through the learning process, and is therefore able to obtain more generalized intermediate features.

[0129] (Modification) The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0130] The inference device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0131] Note that all or part of the functions of the learning device 1 and the inference device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0132] The control unit 21 is an example of an inference unit.

[0133] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0134] REFERENCE SYSTEM 100 1 Learning device 2 Inference device 11 Control unit 12 Interface unit 13 Memory unit 21 Control unit 22 Interface unit 23 Memory unit 91 Processor 92 Memory 93 Processor 94 Memory

Claims

1. A learning device comprising: a control unit that learns a learning object; wherein the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel spectrogram extractor; and the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor.

2. An inference device comprising: a control unit that learns a learning object, wherein the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel spectrogram extractor, and the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor, and an inference unit that performs inference using the mathematical model obtained by the learning device after the learning by the learning device.

3. A learning method executed by a learning device, comprising: a control unit that learns a learning object, wherein the learning involves learning a learning model that is a mathematical model different from PRVAE-VC in that a HuBERT model is used instead of a mel spectrogram extractor, and the learning object is a mathematical model that is a part of the learning model, and is a mathematical model that differs from the mathematical model obtained by PRVAE-VC as the mathematical model used in the inference stage in that a HuBERT model is used instead of a mel spectrogram extractor, the learning method comprising: a learning step in which the control unit performs the learning.

4. A program for causing a computer to function as the learning device according to claim 1.

Images