Learning device, voice conversion device, learning method, voice conversion method, and program

Abstract

One aspect of the present invention is a learning device comprising a control unit that executes a first encoding process in which a processing target is voice data and the processing target is encoded, a content storage conversion process in which content storage conversion is carried out on the voice data, a second encoding process in which modified data, which is the voice data converted by the content storage conversion process, is encoded, and a decoding process in which a result of the first encoding process is decoded, wherein the control unit updates the content of the first encoding process, the second encoding process, and the decoding process so as to increase the degree of matching between the processing target of the first encoding process and a result of the decoding process and the degree of matching between the result of the first encoding process and a result of the second encoding process, on the basis of the processing target of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process.

Description

Learning device, voice conversion device, learning method, voice conversion method and program

[0001] The present invention relates to a learning device, a voice conversion device, a learning method, a voice conversion method, and a program.

[0002] There is growing interest in voice conversion technology using machine learning techniques.

[0003] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).Hsu, Chin-Cheng, et al. "Voice conversion from non-parallel corpora using variational auto-encoder." 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA). IEEE, 2016.Luong, Manh, and Viet Anh Tran. "Many-to-many voice conversion based feature disentanglement using variational autoencoder." arXiv preprint arXiv:2107.06642 (2021).Tobing, Patrick Lumban, et al. "Non-parallel voice conversion with cyclic variational autoencoder." arXiv preprint arXiv:1907.10185 (2019).Kameoka, Hirokazu, et al. "ACVAE-VC: Non-parallel voice conversion with auxiliary classifier variational autoencoder." IEEE / ACM Transactions on Audio, Speech, and Language Processing 27.9 (2019): 1432-1443.Morise, Masanori, Fumiya Yokomori, and Kenji Ozawa. "WORLD: a vocoder-based high-quality speech synthesis system for real-time applications." IEICE TRANSACTIONS on Information and Systems 99.7 (2016): 1877-1884.Jaitly, Navdeep, and Geoffrey E. Hinton. "Vocal tract length perturbation (VTLP) improves speech recognition." Proc. ICML Workshop on Deep Learning for Audio, Speech and Language. Vol. 117. 2013.Kurematsu, Akira, et al. "ATR Japanese speech database as a tool of speech recognition and synthesis." Speech communication 9.4 (1990): 357-363.

[0004] However, in speech conversion technology, the speaker to be converted is not necessarily the same in the training and inference stages, as training is performed using a first speaker, and then the speech of a second speaker different from the first speaker is converted in the inference stage. As a result, the accuracy of conversion in the inference stage may be low.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for improving the accuracy of voice conversion.

[0006] One aspect of the present invention is a learning device that includes a control unit that executes a first encoding process that encodes audio data as an object to be processed, a content-preserving conversion process that performs a content-preserving conversion on the audio data, a second encoding process that encodes modified data that is the audio data after conversion by the content-preserving conversion process, and a decoding process that decodes a result of the first encoding process, and the control unit updates the contents of the first encoding process, the second encoding process, and the decoding process based on the object to be processed of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process, so as to increase the degree of agreement between the object to be processed of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process.

[0007] One aspect of the present invention is a voice conversion device comprising: a control unit that executes a first encoding process that encodes voice data as a processing object; a content-preserving conversion process that performs content-preserving conversion on the voice data; a second encoding process that encodes modified data that is the voice data after conversion by the content-preserving conversion process; and a decoding process that decodes the result of the first encoding process, wherein the control unit updates the contents of the first encoding process, the second encoding process, and the decoding process based on the processing object of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process so as to increase the degree of agreement between the processing object of the first encoding process and the result of the decoding process, and the degree of agreement between the result of the first encoding process and the result of the second encoding process; and a voice conversion unit that performs voice conversion using the first encoding process and the decoding process obtained by a learning device when a predetermined termination condition is satisfied.

[0008] One aspect of the present invention is a learning method that includes a control step of executing a first encoding process that encodes audio data as an object to be processed, a content-preserving conversion process that performs content-preserving conversion on the audio data, a second encoding process that encodes modified data that is the audio data after conversion by the content-preserving conversion process, and a decoding process that decodes a result of the first encoding process, wherein the control step updates the contents of the first encoding process, the second encoding process, and the decoding process based on the object to be processed of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process, so as to increase the degree of agreement between the object to be processed of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process.

[0009] One aspect of the present invention is a voice conversion method comprising: a control step of executing a first encoding process that encodes voice data as a processing object; a content-preserving conversion process that performs content-preserving conversion on the voice data; a second encoding process that encodes modified data that is the voice data after conversion by the content-preserving conversion process; and a decoding process that decodes a result of the first encoding process, wherein the control step updates contents of the first encoding process, the second encoding process, and the decoding process based on the processing object of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process so as to increase the degree of match between the processing object of the first encoding process and the result of the decoding process and the degree of match between the result of the first encoding process and the result of the second encoding process; and a voice conversion step of performing voice conversion using the first encoding process and the decoding process at the time when a predetermined termination condition obtained by a learning method is satisfied.

[0010] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0011] One aspect of the present invention is a program for causing a computer to function as the above-described voice conversion device.

[0012] The present invention makes it possible to improve the accuracy of voice conversion.

[0013] FIG. 1 is an explanatory diagram illustrating a voice conversion system according to an embodiment. FIG. 2 is a diagram illustrating an example of a more specific hardware configuration of a learning device according to an embodiment. FIG. 3 is a diagram illustrating an example of a processing flow executed by a learning device according to an embodiment. FIG. 4 is a diagram illustrating an example of a more specific hardware configuration of a voice conversion device according to an embodiment. FIG. 5 is a diagram illustrating an example of a processing flow executed by a voice conversion device according to an embodiment. FIG. 6 is an explanatory diagram illustrating experimental conditions according to an embodiment. FIG. 7 is a diagram illustrating a first example of experimental results according to an embodiment. FIG. 8 is a diagram illustrating a second example of experimental results according to an embodiment. FIG. 9 is a diagram illustrating a third example of experimental results according to an embodiment.

[0014] 1 is an explanatory diagram illustrating a voice conversion system 100 according to an embodiment. The voice conversion system 100 includes a learning device 1 and a voice conversion device 2.

[0015] The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus. The control unit 11 executes learning processing.

[0016] In the learning process, the control unit 11 executes a first encoding process, a content preserving conversion process, a second encoding process, and a decoding process. The first encoding process is a process for encoding a processing target, which is audio data.

[0017] The content-preserving conversion process is a process of performing content-preserving conversion on a processing target. The processing target of the content-preserving conversion process is the processing target of the first encoding process. Therefore, the processing target of the content-preserving conversion is audio data.

[0018] The second encoding process encodes the audio data (hereinafter referred to as "corrected data") after conversion by the content-preserving conversion process. The decoding process decodes the result of the first encoding process. The decoding process may also use input speaker information indicating the speaker of the audio data processed by the first encoding process.

[0019] In the learning process, the control unit 11 updates the contents of the first encoding process, the second encoding process, and the decoding process based on the processing target of the first encoding process, the result of the decoding process, and the results of the first encoding process and the second encoding process so as to increase the self-consistency and the cross-consistency. The self-consistency is the degree of agreement between the processing target of the first encoding process and the result of the decoding process. The cross-consistency is the degree of agreement between the result of the first encoding process and the result of the second encoding process.

[0020] The updating of the contents of the first encoding process, the second encoding process, and the decoding process is performed until a predetermined update termination condition (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition may be, for example, that a predetermined number of updates have been performed, or that the difference between the contents of the first encoding process, the second encoding process, and the decoding process before and after the update is smaller than a predetermined difference.

[0021] The voice conversion device 2 converts the voice using the first encoding process and the first decoding process when the learning end condition is satisfied.

[0022] <Effects of the learning process executed by the control unit 11> As described above, the learning process executes a first encoding process, a content-preserving conversion, a second encoding process, and a decoding process, and updates the first encoding process, the second encoding process, and the decoding process to increase the self-consistency and the cross-species matching. As described above, the self-consistency indicates how well the encoded data can be restored, and the cross-species matching indicates the degree of matching between the encoding results of the audio data generated by the content-preserving conversion and the audio data that was the source of the content-preserving conversion.

[0023] As is well known, content-preserving conversion is a process that linearly converts, for example, volume, spectral envelope, or fundamental frequency. Therefore, content-preserving conversion is a conversion that preserves linguistic information. Therefore, content-preserving conversion generates speech data that has different paralinguistic information but identical linguistic information compared to the original speech data. This means that pseudo-speech data from a different speaker is generated.

[0024] The audio data converted by this content-preserving conversion is encoded by a second encoding process. On the other hand, the source audio data of the content-preserving conversion is encoded by a first encoding process. The encoding is a process for extracting features. Therefore, to increase the degree of agreement between the results of the first encoding process and the second encoding process, the processes are updated so that the amount of paralinguistic information in both the results of the first encoding process and the second encoding process is reduced and only linguistic information is extracted.

[0025] One of the reasons for the decrease in accuracy in speech conversion when the speaker is different from that used during training is that speech conversion that is highly dependent on information about the original speaker is learned during training. Therefore, if a conversion that is less dependent on information about the original speaker is learned during training, the accuracy of speech conversion will improve. In other words, the less speaker-dependent the latent space variables become through training, the more accurate the speech conversion will be.

[0026] In the learning process performed by the control unit 11, speech data in which speaker information is pseudo-converted is generated by content-preserving conversion, and then learning is performed to increase the degree of cross-species matching. Learning that increases the degree of cross-species matching is performed by learning such that the results of the first encoding process and the second encoding process contain as much linguistic information as possible. In other words, learning that increases the degree of cross-species matching is learning that reduces the speaker dependency of latent space variables.

[0027] As a result, speech conversion can be performed with high accuracy even when converting speech from a speaker different from that used during training. In other words, the training process improves the accuracy of speech conversion. Note that the training process is performed to increase not only the degree of cross-species agreement but also the degree of self-agreement, so the result of the first-type encoding process at the time the training termination condition is met is not a process of acquiring only linguistic information.

[0028] The first encoding process is, for example, a process performed by an encoder in a variational autoencoder, and in this case, the decoding process is a process performed by a decoder in the variational autoencoder.

[0029] In the learning process, modified speaker information indicating the speaker of the modified data may be used. Since the speaker of the modified data is the speaker to which the data is converted in the content-preserving conversion process, the modified speaker information also indicates the speaker to which the data is converted in the content-preserving conversion process. The modified speaker information is input, for example, from outside the learning device 1 by something other than the learning device 1, such as a user.

[0030] The corrected speaker information may be acquired based on the correction data, for example. Acquisition based on the correction data is a process of estimating whether a speaker is male or female based on the range of the voice data indicated by the correction data, and acquiring information indicating the result of the estimation as the corrected speaker information.

[0031] The modified speaker information is acquired in the second encoding process. The second encoding process, having acquired the modified speaker information, encodes the processing target based on the modified speaker information as well. The use of modified speaker information in encoding allows for more accurate estimation of non-linguistic information than when the modified speaker information is not used, thereby further increasing the degree of heterogeneous matching. As a result, the use of modified speaker information enables more accurate speech conversion than when the modified speaker information is not used.

[0032] <Example of Objective Function Used in Learning Process> Here, an example of the objective function used in the learning process will be described using a mathematical expression. In the learning process, for example, the following objective function L is used.

[0033]

[0034] Here, the first term on the right side represents the self-consistency. ｐｒ represents the degree of heterogeneity. ｐｒ is a predetermined weighting factor.

[0035] Here, taking the case where the first encoding process and the decoding process are respectively processes by an encoder and a decoder of a variational autoencoder (VAE), the self-consistency L ｃａｖｅ and the heterogeneity match L ｐｒ We derive an example of a mathematical expression for

[0036] <Derivation of an example of mathematical expression of self-consistency> VAE is a neural network model including an encoder network and a decoder network (see, for example, Non-Patent Document 1). The encoder network derives the conditional distribution q of the latent space variable z when conditioned by data x and attribute code c. φ Generate the distribution parameters of (z|x, c).

[0037] On the other hand, the decoder network calculates the conditional distribution p of data x when the latent space variable z and the attribute code c are given. θ Generate distribution parameters of (x|z, c). The log marginal likelihood of data x when attribute code c is given is given by the following equation (2).

[0038]

[0039] Here, KL[·|·] represents the Kullback-Leibler (KL) divergence. KL divergence is the process of maximizing L(θ,φ) with respect to θ and φ, so that q φ We show that the KL distance between (z|x, c) and p(z) can be minimized. φ (z|x, c) and p θ A Gaussian distribution is often assumed to model (x|z, c).

[0040] In VAE-based speech conversion (see, for example, Non-Patent Document 2), the encoder and decoder use a mel-spectrogram x ｓ and the speaker vector c of the input speaker ｓ When is given, as expressed by equations (3) to (5), q φ and p θ A distribution parameter series is generated. The distribution parameter series is a series of the mean and logarithmic variance.

[0041]

[0042]

[0043]

[0044] ε represents Gaussian noise. [·;·] represents coupling in the channel dimension. The symbols in the following equation (6) represent element products. μ ｚｓ means the mean value sequence of the latent space variables for the input speech. σ ２ ｚｓ denotes the variance value sequence of the latent space variables for the input speech. Encoder(·) denotes the encoding process in the VAE. μ ｘｓｓ means the mean value sequence of the reconstructed features of the input speech. Decoder(·) means the decoding process in the VAE. σ ｚｓ means the standard deviation value sequence of the latent space variables for the input speech. ｓｓ means the reconstructed feature of the input speech. σ ｘｓｓ means the variance value sequence of the reconstructed feature of the input speech.

[0045]

[0046] In the training process, the speaker vector c of the target speaker ｔ Then, the transformed mel spectrogram x ｓｔ is generated as expressed by the following equations (7) and (8).

[0047]

[0048]

[0049] μ ｘｓｔ means the average value sequence of the features of the converted speech. ２ ｘｓｔ means the average value sequence of the features of the converted speech.

[0050] From this, the objective functions to be minimized are functions expressed by the following equations (9) to (11).

[0051]

[0052]

[0053]

[0054] N(·) represents the Gaussian distribution. ＧＮＬＬ represents the loss function for the negative log-likelihood.ＫＬＤ represents the loss function for the KL term.

[0055] <Derivation of an example of mathematical expression of different-type matching degree> First, an example of generating the above-mentioned correction data will be described. Here, as an example, a case where speaker biases such as the average value of the fundamental frequency F0 of the speech data to be converted and the vocal tract length are manipulated will be described. ａｎａ and WORLD Synthesis F ｓｙｎ and F0 and f ｓ and the spectral envelope e ｓ and the non-periodic component a ｓ and after extracting and manipulating the extracted acoustic features, the corrected data w ｍ The process of generating the above is expressed by the following equations (12) and (13) (see, for example, Non-Patent Document 6).

[0056]

[0057]

[0058] F ｆ０ represents a function that controls the average value of the fundamental frequency F0. ｆ represents the target mean value of F0. ｅｎｖ represents the frequency warping function (see, for example, Non-Patent Document 7). ｅ represents the frequency warping factor.

[0059] After generating the corrected data, the second encoding process uses the above-mentioned x ｓ As with obtaining ｍ Mel spectrogram x ｍ Then, in the second encoding process, the speaker vector c is obtained from the mel spectrogram of the corrected data through a speaker encoder. ｍ Next, in the second encoding process, the latent space variable z ｍ The distribution parameter series μ of the conditional distribution of ｚｍ and σ ｚｍ and generate the Mel spectrogram x ｍ Such second encoding processing after acquisition is expressed by the following equations (14) and (15).

[0060]

[0061]

[0062] SpeakerEncoder represents a speaker encoder. When the encoding and decoding processes expressed by equations (2) to (15) are executed, the heterogeneous matching degree is calculated by the latent space variable z ｓ and Z ｍ Therefore, the heterogeneous coincidence degree is expressed by, for example, the following equation (16):

[0063]

[0064] 2 is a diagram showing an example of a more specific hardware configuration of the learning device 1 in an embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU and a memory 92 connected by a bus as described above, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0065] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0066] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a learning process. The control unit 11 controls, for example, the operation of the interface unit 12, and transmits and receives information to and from external devices via the interface unit 12. Therefore, the control unit 11 acquires data to be used for learning, for example, via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0067] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data used for learning. In such a case, the interface unit 12 receives the data used for learning by communicating with the device that transmits the data used for learning. The external device is, for example, the voice conversion device 2.

[0068] In such a case, the interface unit 12 communicates with the speech conversion device 2 to transmit to the speech conversion device 2 a computer program (hereinafter referred to as the "trained program") that executes the first encoding process and the trained decoding process at the time when the learning termination condition is met (hereinafter referred to as "trained"). Having acquired the trained program, the speech conversion device 2 performs speech conversion by executing the trained program. The external device may be, for example, the device that transmitted the input speaker information. In such a case, the interface unit 12 acquires the input speaker information by communicating with the device that transmitted the input speaker information.

[0069] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the study device 1. In this way, the interface unit 12 accepts input of various information to the study device 1 via the input device, either wired or wireless.

[0070] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0071] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 stores information acquired by the interface unit 12, for example.

[0072] 3 is a diagram showing an example of the flow of processing executed by the learning device 1 in this embodiment. The control unit 11 acquires voice data via the interface unit 12, etc. (Step S101). Next, the control unit 11 executes a first encoding process on the voice data acquired in Step S101 (Step S102).

[0073] Next, the control unit 11 performs a content-preserving conversion process on the voice data obtained in step S101 (step S103). Next, the control unit 11 performs a second encoding process on the result of the content-preserving conversion process in step S103 (step S104). Next, the control unit 11 performs a decoding process on the result of the first encoding process in step S102 (step S105). The decoding process may use input speaker information.

[0074] Next, the control unit 11 determines whether the learning termination condition is satisfied (step S106). If the learning termination condition is satisfied (step S106: YES), the process ends. On the other hand, if the learning termination condition is not satisfied (step S106: NO), the control unit 11 updates the contents of the first encoding process, the decoding process, and the second encoding process so as to increase the self-consistency and the heterogeneous consistency (step S107). After step S107, the process returns to step S101.

[0075] Note that the processes of steps S102 to S105 may be executed at any timing as long as they are all executed after step S101 and before step S106, the process of step S104 is executed after step S103, and the process of step S105 is executed after step S102.

[0076] 4 is a diagram showing an example of a more specific hardware configuration of the voice conversion device 2 in the embodiment. The voice conversion device 2 includes a control unit 21 including a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. By executing the program, the voice conversion device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0077] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the voice conversion device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0078] The control unit 21 controls the operation of each functional unit included in the voice conversion device 2. The control unit 21 executes, for example, a trained program. That is, the control unit 21 executes, for example, a trained first encoding process and a trained decoding process.

[0079] The control unit 21 controls, for example, the operation of the interface unit 22. The control unit 21 acquires, for example, audio data via the interface unit 22. The control unit 21 acquires, for example, information stored in the storage unit 23. Specifically, the process of acquiring the information stored in the storage unit 23 is reading.

[0080] The interface unit 22 includes a communication interface for connecting the voice conversion device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, the learning device 1. In this case, the interface unit 22 obtains the learned program by communicating with the learning device 1. The external device is, for example, a device that is the source of voice data. In this case, the interface unit 22 obtains voice data by communicating with the device that is the source of voice data.

[0081] The interface unit 22 may be configured to include input devices such as a mouse, a keyboard, a touch panel, etc. The interface unit 22 may be configured as an interface that connects these input devices to the voice conversion device 2. In this way, the interface unit 22 accepts input of various information to the voice conversion device 2 via the input devices, either wired or wireless.

[0082] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the voice conversion device 2. The interface unit 22 outputs, for example, information input to the interface unit 22.

[0083] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the voice conversion device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 stores information acquired by the interface unit 22, for example.

[0084] 5 is a diagram showing an example of the flow of processing executed by the voice conversion device 2 in the embodiment. For simplicity, in FIG. 5, an example will be described in which the voice conversion device 2 has acquired a trained program in advance.

[0085] Voice data is input to the interface unit 22, and the control unit 21 acquires the voice data (step S201). Next, the control unit 21 executes the learned program on the voice data acquired in step S201 (step S202).

[0086] That is, the control unit 21 performs a trained first encoding process on the voice data acquired in step S201, and then performs a trained decoding process to convert the voice data into voice data of the voice of the speaker indicated by the target speaker information. The target speaker information is information indicating the speaker to which the voice is to be converted (the so-called target speaker). The target speaker information may be stored in advance in the storage unit 23, or may be input by an external entity such as a user via the interface unit 22. In this way, voice conversion is performed by executing step S202.

[0087] <Experimental Results> An example of an experiment using the voice conversion system 100 and the results thereof will be described. In the experiment, the technology described in Non-Patent Document 2 (hereinafter referred to as "CVAE"), the technology described in Non-Patent Document 3 (hereinafter referred to as "+beta"), the technology described in Non-Patent Document 4 (hereinafter referred to as "+cc"), and the technology described in Non-Patent Document 5 (hereinafter referred to as "+ac") were used as comparative technologies for evaluating the performance of the voice conversion system 100.

[0088] More specifically, in the experiment, two different sizes of mathematical models were used for each of CAVE, +beta, +cc, and +ac. Two different sizes of mathematical models were also used for the voice conversion system 100. Details of the sizes, etc., are as shown in FIG. 6 below.

[0089] FIG. 6 is an explanatory diagram illustrating experimental conditions in an embodiment. FIG. 6 shows, for example, that LSTM (Long Short Term Memory) was used for both the encoder and decoder in the experiment. In FIG. 6, "Small" indicates the name of a small-sized model, "Large" indicates the name of a large-sized model, "Projection" indicates the fully connected layer, "dim." indicates the number of output dimensions, "layer" indicates the number of layers, "residual" indicates the residual connection, and "Num. of Params" indicates the number of learnable parameters of the model. Therefore, for example, the number 512 in area A1 indicates the number of output dimensions of the fully connected layer. The number 4 in area A2 indicates the number of layers of the LSTM. The number 512 in area A3 indicates the number of output dimensions of the LSTM. The number in area A4 indicates the number of output dimensions of the fully connected layer.

[0090] The number 512 in area A5 means the number of dimensions of the output of the fully connected layer. The number 4 in area A6 means the number of layers of the LSTM. The number 512 in area A7 means the number of dimensions of the output of the LSTM. The number in area A8 means the number of dimensions of the output of the fully connected layer. The number in area A9 means the number of learnable parameters of the Small model.

[0091] In the experiment, 450 sentences out of 503 sentences in the Japanese speech dataset ATR phoneme-balanced sentences (see, for example, Non-Patent Document 8) were used as training data and 53 sentences as evaluation data. The experiment included three objective evaluation experiments: an experiment to obtain mel-cepstral distortion, an experiment to obtain the correlation coefficient of log F0, and an experiment to obtain the speech recognition error rate, as well as two listening experiments: an experiment to obtain sound quality and an experiment to obtain speaker similarity.

[0092] The audio sampling rate of the audio data used in the experiment was 16 kHz. Note that the audio data used in the experiment was audio data from which silent periods had been removed in advance.

[0093] In the experiment, dynamic time warping (DTW) was used to calculate Mel-Cepstral Distortion (MCD) and the logarithm F0 correlation coefficient (F0Corr) to obtain alignment between the converted speech and the reference speech. A lower MCD value indicates less distortion and better performance. A higher F0Corr value indicates better intonation performance. In the experiment, the Character Error Rate (CER) was calculated as the speech recognition error rate. A lower CER value indicates better quality.

[0094] Voice quality was evaluated using the Mean Opinion Score (MOS) Test. The MOS rating is on a 5-point scale, with 5 being the best and 1 being the worst. Therefore, the higher the MOS value, the better the voice quality.

[0095] Speaker similarity was evaluated using the Preference Test (PT). PT ratings are on a four-point scale from 1 to 4, with 4 representing same (sure), 3 representing same (not sure), 2 representing different (not sure), and 1 representing different (sure). Therefore, the higher the percentage of same speakers, the higher the similarity and the better the performance.

[0096] FIG. 7 is a diagram showing a first example of experimental results in an embodiment. In FIG. 7, the technologies listed in the rows above "CAVE-Large" are technologies using small-sized models. Therefore, for example, "CAVE-Small" is a small-sized CAVE. For example, "+beta" listed immediately below "CAVE-Small" is a small-sized +beta. On the other hand, the technologies listed in the rows below "CAVE-Large" in FIG. 7 are technologies using large-sized models.

[0097] The less speaker-dependent the latent space variables are, the smaller the difference between the reconstruction error when converting from an input speaker to another (hereinafter referred to as "reconstruction") and the conversion error when converting from an input speaker to a target speaker (hereinafter referred to as "evaluation target difference") should be. The reconstruction error is calculated by dividing the reconstructed speech x ｓｓ and input voice x ｓ The conversion error is the difference between the converted speech x ｓｔ and x ｓ and the target speech, which is the same speech but spoken by the target speaker.

[0098] Therefore, in the speech conversion system 100, the reconstruction error is the difference between the speech data obtained by auto-encoding the speech data of the input speaker's speech using the trained first encoding process and the trained decoding process (therefore, the speaker is the speech data of the input speaker's speech) and the speech data before auto-encoding. In the speech conversion system 100, the conversion error is the difference between the speech data obtained by conversion and the speech data of the target speech when the speech data of the input speaker's speech is converted into speech data of the target speaker's speech using the trained first encoding process and the trained decoding process.

[0099] Therefore, Figure 7 shows the evaluation target differences. In Figure 7, the numbers in parentheses represent the evaluation target differences. In Figure 7, "+pr(ours)" represents the speech conversion system 100. Therefore, Figure 7 shows that for the small-sized speech conversion system 100, the evaluation target differences are 0.78 for MCD, 0.10 for F0Corr, and 1.3 for CER.

[0100] 7 also shows that for the larger size speech conversion system 100, the evaluation target difference is 1.00 for MCD, 0.11 for F0Corr, and 1.4 for CER. Figure 7 shows that regardless of size, speech conversion system 100 has smaller evaluation target differences than the other comparison technologies, whether it is MCD, F0Corr, or CER.

[0101] Thus, Figure 7 shows that the speech conversion system 100 performs speech conversion with higher accuracy than other comparable techniques. Figure 7 also shows that the evaluation target difference is smaller for larger sizes than for smaller sizes.

[0102] FIG. 8 is a diagram showing a second example of experimental results in the embodiment. FIG. 9 is a diagram showing a third example of experimental results in the embodiment. The horizontal axes of FIGS. 8 and 9 indicate each technology, including the technology being compared. Technologies to the left of "CAVE-Large" on the horizontal axes of FIGS. 8 and 9 are technologies that use small-sized models. Technologies to the right of "CAVE-large," including "CAVE-Large" on the horizontal axes of FIGS. 8 and 9, are technologies that use large-sized models.

[0103] In the examples of Figures 8 and 9, "+pr(ours)" also represents the voice conversion system 100. The vertical axis of Figure 8 represents the MOS value. The vertical axis of Figure 9 represents the ratio of the evaluation value. "Re-synthesized" in Figure 8 means analytically resynthesized sound (a certain upper limit value of voice conversion).

[0104] The results in Figures 8 and 9 show that when the model size is large, the speech conversion system 100 outperforms the compared technologies in terms of sound quality and speaker similarity. The results in Figures 8 and 9 also show that when the model size is small and the model's expressive power is weak, the difference between the compared technologies and between the speech conversion system 100 and the compared technologies is small, but as the model size increases, the conversion performance of the compared technologies deteriorates and that of the speech conversion system 100 improves.

[0105] The learning device 1 configured in this manner executes the first encoding process, content-preserving conversion, second encoding process, and decoding process, and updates the first encoding process, second encoding process, and decoding process so as to increase the self-consistency and cross-species consistency (i.e., to reduce the speaker dependency of the latent space variables). Therefore, the learning device 1 can improve the accuracy of speech conversion, as described above in <Effects of the learning process executed by the control unit 11>.

[0106] The speech conversion device 2 configured in this manner performs speech conversion by executing the first encoding process and the learned decoding process obtained by the learning device 1. Therefore, the speech conversion device 2 can improve the accuracy of speech conversion.

[0107] The voice conversion system 100 configured in this manner includes the learning device 1. Therefore, the voice conversion system 100 can improve the accuracy of voice conversion.

[0108] (Modification) The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0109] The voice conversion device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0110] Note that all or part of the functions of the learning device 1 and the voice conversion device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may also be transmitted via telecommunications lines.

[0111] The control unit 21 is an example of a voice conversion unit.

[0112] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0113] 100... speech conversion system, 1... learning device, 2... speech conversion device, 11... control unit, 22... interface unit, 23... storage unit, 21... control unit, 22... interface unit, 23... storage unit, 91... processor, 92... memory, 93... processor, 94... memory

Claims

1. A learning device comprising: a control unit that executes a first encoding process that treats audio data as a processing object and encodes the processing object; a content-preserving conversion process that performs a content-preserving conversion on the audio data; a second encoding process that encodes modified data, which is the audio data after conversion by the content-preserving conversion process; and a decoding process that decodes a result of the first encoding process, wherein the control unit updates the contents of the first encoding process, the second encoding process, and the decoding process based on the processing object of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process, so as to increase the degree of agreement between the processing object of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process.

2. The learning device according to claim 1, wherein the first encoding process is a process performed by an encoder in a variational autoencoder, and the decoding process is a process performed by a decoder in the variational autoencoder.

3. The learning device according to claim 1, wherein the second encoding process encodes the processing target also based on corrected speaker information indicating a speaker of the corrected data.

4. A voice conversion device comprising: a control unit that executes a first encoding process that treats voice data as a processing object and encodes the processing object, a content-preserving conversion process that performs content-preserving conversion on the voice data, a second encoding process that encodes modified data, which is the voice data after conversion by the content-preserving conversion process, and a decoding process that decodes a result of the first encoding process, wherein the control unit updates the contents of the first encoding process, the second encoding process, and the decoding process based on the processing object of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process, so as to increase the degree of agreement between the processing object of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process; and a voice conversion unit that performs voice conversion using the first encoding process and the decoding process obtained by a learning device at the point when a predetermined termination condition is satisfied.

5. A learning method comprising a control step of executing a first encoding process which treats audio data as an object to be processed and encodes the object to be processed, a content-preserving conversion process which performs a content-preserving conversion on the audio data, a second encoding process which encodes modified data which is the audio data after conversion by the content-preserving conversion process, and a decoding process which decodes a result of the first encoding process, wherein in the control step, contents of the first encoding process, the second encoding process, and the decoding process are updated based on the object to be processed of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process so as to increase the degree of agreement between the object to be processed of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process.

6. A voice conversion method comprising: a control step of executing a first encoding process which encodes a processing object which is voice data; a content-preserving conversion process which performs a content-preserving conversion on the voice data; a second encoding process which encodes modified data which is the voice data after conversion by the content-preserving conversion process; and a decoding process which decodes a result of the first encoding process, wherein in the control step, contents of the first encoding process, the second encoding process, and the decoding process are updated based on the processing object of the first encoding process, the result of the decoding process, the result of the first encoding process, and the result of the second encoding process so as to increase the degree of agreement between the processing object of the first encoding process and the result of the decoding process and the degree of agreement between the result of the first encoding process and the result of the second encoding process; and a voice conversion step of performing voice conversion using the first encoding process and the decoding process at the point in time when a predetermined termination condition obtained by the learning method is satisfied.

7. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

8. A program for causing a computer to function as the voice conversion device according to claim 4.

Images