Learning device, voice conversion system, learning method, and program

The voice conversion system addresses the issue of suboptimal performance by learning a voice conversion model and speaker authentication model using GE2E loss, resulting in accurate and high-quality voice conversion that effectively represents the target speaker's voice.

WO2024247002A1PCT designated stage expired Publication Date: 2024-12-05NT T INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/019764
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional voice conversion methods often fail to accurately reflect the characteristics of the target speaker during the conversion process, leading to suboptimal performance in voice quality conversion.

Method used

A voice conversion system that learns a voice conversion model and a speaker authentication model simultaneously using Generalized End-to-End (GE2E) loss, incorporating speaker embedding expressions to improve the accuracy and performance of voice quality conversion.

Benefits of technology

The system achieves higher quality voice conversion by ensuring the converted speech faithfully represents the target speaker's characteristics, as demonstrated by improved objective and subjective evaluation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023019764_05122024_PF_FP_ABST
    Figure JP2023019764_05122024_PF_FP_ABST
Patent Text Reader

Abstract

One aspect of the present invention is a learning device comprising: a voice conversion model for converting a voice signal to a voice signal of another person; a speaker authentication model for authenticating a speaker; and a learning unit that uses the voice signals of a plurality of the speakers and identification information for identifying the speakers to learn a learning criterion regarding the speaker authentication model based on GE2E loss simultaneously with the voice conversion model.
Need to check novelty before this filing date? Find Prior Art

Description

Learning device, voice conversion system, learning method and program

[0001] The present invention relates to a learning device, a voice conversion system, a learning method, and a program technology.

[0002] Voice conversion is a technology that converts non-linguistic and paralinguistic information contained in the input speech signal while preserving the linguistic information. It is useful for a variety of tasks, such as text-to-speech synthesis, speech recognition, and speech assistance and augmentation. There are various approaches to achieving voice conversion, but they can be broadly categorized into parallel conversion approaches and non-parallel approaches depending on the constraints on the training data required to train the conversion model. In the parallel conversion approach, parallel corpora in which the input and target speakers produce identical utterances are prepared for training the conversion model, and a mapping function expressed as a Gaussian mixture model or a deep neural network is trained.

[0003] On the other hand, non-parallel conversion approaches have attracted attention in recent years because training data is not limited to parallel corpora and data collection is easy. In non-parallel speech conversion methods, two approaches have been proposed from the perspective of training methods: generative adversarial networks (GANs) and variational autoencoders (VAEs). In recent years, non-parallel speech conversion approaches have been proposed that enable conversion between arbitrary speakers, in addition to conversion between pre-trained input and target speakers. A representative example of this is a method called AutoVC (see, for example, Non-Patent Document 1). AutoVC is an advanced version of the VAE-based non-parallel speech conversion approach, and uses a speech conversion model with an autoencoder structure and a pre-trained speaker recognition model. The speaker recognition model utilizes a previously proposed speaker authentication model based on generalized end-to-end (GE2E) loss, which is pre-trained separately from the training of the conversion model. The speech conversion model is then trained using a reconstruction error-based training criterion, with the speaker embeddings obtained from the trained speaker verification model as the conditions for conversion. By using this framework that utilizes both the speech conversion model and the speaker verification model, non-parallel speech conversion between any speakers becomes possible, and its effectiveness has been demonstrated.

[0004] Kaizhi Qian, Yang Zhang, Shiyu Chang, et al, "AutoVC:Zero-Shot Voice Style Transfer with Only Autoencoder Loss", in International Conference on Machine Learning, pp. 5210-5219, 2019

[0005] However, in the prior art, the conversion process during inference is not taken into account when training a voice conversion model, which can result in cases where the converted speech does not faithfully reflect the characteristics of the target speaker, or where a bottleneck in conversion performance occurs. In view of the above circumstances, the present invention aims to provide a technology that can improve the accuracy and performance of voice conversion.

[0006] One aspect of the present invention is a learning device that includes a voice conversion model that converts a voice signal into the voice signal of another person, a speaker authentication model that authenticates speakers, and a learning unit that uses voice signals of multiple speakers and identification information that identifies the speakers to learn a learning criterion for the speaker authentication model based on GE2E (Generalized End-to-End) loss simultaneously with the voice conversion model.

[0007] One aspect of the present invention is a voice conversion system comprising the above-mentioned learning device, the trained voice conversion model, and a voice conversion device that converts a voice signal into a voice signal of another person using the trained speaker authentication model.

[0008] One aspect of the present invention is a training method for a training device having a voice conversion model that converts a voice signal into the voice of another person and a speaker authentication model that authenticates speakers, in which a training unit uses voice signals of multiple speakers and identification information that identifies the speakers to train a learning criterion for the speaker authentication model based on GE2E (Generalized End-to-End) loss simultaneously with the voice conversion model.

[0009] One aspect of the present invention is a program for causing a computer to function as the learning device.

[0010] According to the present invention, it is possible to improve the accuracy and performance of voice conversion.

[0011] FIG. 1 is a diagram showing an example of the configuration of a voice conversion system according to an embodiment; FIG. 2 is a diagram showing an example of a learning process procedure performed by a learning device according to an embodiment; FIG. 3 is a diagram showing an example of a voice conversion process procedure performed by a voice conversion device according to an embodiment; FIG. 4 is a diagram showing an example of an objective evaluation result; FIG. 5 is a diagram showing an example of a subjective evaluation result; FIG. 6 is a diagram showing an example of an acoustic feature sequence of a voice signal of an input speaker; FIG. 7 is a diagram showing an example of an acoustic feature sequence of a voice signal converted by an AutoVC method which is a conventional technology; FIG. 8 is a diagram showing an example of an acoustic feature sequence of a voice signal of a target speaker; and FIG. 9 is a diagram showing an example of an acoustic feature sequence of a voice signal converted by the method of an embodiment.

[0012] Embodiments of the present invention will be described in detail with reference to the drawings. First, an overview of the conventional technologies AutoVC and ACVAE-VC will be described. AutoVC (see Non-Patent Document 1) is a non-parallel speech conversion method based on an autoencoder, and is composed of a speech conversion model consisting of an encoder and a decoder, and a pre-trained speaker authentication model. The speech conversion model is trained to minimize the reconstruction-based training criterion defined by the following equations (1) to (3), using network parameters φ', θ', and Ψ' of the encoder, decoder, and speaker authentication model. Note that equations (1) and (2) represent errors in the feature space, and equation (3) represents the reconstruction error in the embedding space.

[0013]

[0014]

[0015]

[0016] In equations (1) to (3), X is the acoustic feature sequence of speech, y is the label representing speaker information, and f φ’ (·) is the output of the encoder, f θ’ (·) is the decoder output, f ψ’ (·) represents the output of the speaker authentication model. φ’ (·) represents the intermediate output of the decoder. E (double underlined letter) y~p(y)[·] means the expected value of [·]. p(·|y) represents the conditional probability of y. Also, ||·|| is a norm, representing the distance in a metric space. The subscript number of ||·|| represents the norm value, with "1" being the 1-norm (absolute value norm), "2" being the 2-norm (Euclidean norm), and "3" being the 3-norm. Also, the superscript number "2" of ||·|| represents a square. Also, i.i.d. means independent and identically distributed.

[0017] Next, an outline of ACVAE (Auxiliary Classifier VAE (Variational AutoEncoder))-VC (Voice Conversion) (see, for example, Reference 1) will be given.

[0018] ACVAE-VC is a non-parallel speech conversion method based on VAE, which uses a speech conversion model with an autoencoder structure and a pre-trained speaker recognition model. The speaker recognition model uses a speaker authentication model based on the Generalized End-to-End (GE2E) loss (see, for example, Reference 2). The speech conversion model is trained using a reconstruction error-based training criterion, with speaker embeddings obtained from the trained speaker authentication model as the conversion condition. ACVAE-VC enables non-parallel speech conversion between any speakers by using a framework that utilizes both a speech conversion model and a speaker authentication model.

[0019] In ACVAE-VC, it is assumed that the distributions of the encoder and decoder follow the Gaussian distributions expressed by the following equations (4) and (5), respectively.

[0020]

[0021]

[0022] In equations (4) and (5), μ φ (X, y) and σ φ 2 (X, y) are parameters that represent the encoder distribution. μ θ (Z, y) and σ φ 2(Z, y) are parameters representing the distribution of the decoder. Also, diag(·) is a diagonal matrix. If the network parameters of the encoder and decoder are φ and θ, ACVAE-VC learns to maximize the variational lower bound of the following equation (6) as the learning criterion.

[0023]

[0024] In formula (6), D KL [·||·] is the Kullback-Leivler (KL) divergence. The prior distribution p(Z) is assumed to be a standard normal distribution. Unlike AutoVC and other non-parallel speech conversion methods based on VAE, ACVAE-VC uses the expected value of the mutual information I(y; X|Z) as the learning criterion. As a result, ACVAE-VC can predict the output X~p θ The speech conversion model is trained so that (X|Z, y) reflects the label y as closely as possible. However, since it is difficult to directly use the mutual information, ACVAE-VC uses the variational lower bound of the following equation (7).

[0025]

[0026] In formula (7), r ψ (y|X) represents the distribution of the auxiliary classifier represented by the network parameter ψ, and in the case of speaker conversion, the auxiliary classifier can be regarded as a speaker identification model. In addition, in ACVAE-VC, the cross-entropy of the following equation (8) is also used as a training criterion.

[0027]

[0028] Reference 1: Hirokazu Kameoka, Takuhiro, et al., “ACVAEVC:Non-Parallel Voice Conversion With Auxiliary Classifier Variational Autoencoder”, IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 9, pp.1432-1443, 2019

[0029] Reference 2: G. Heigold, I. Moreno, et al., “End-to-end text-dependent speaker verification”, Proc. ICASSP, IEEE Press, pp.5115-5119, 2016

[0030] (Training Method of the Present Embodiment) Next, the training method of the present embodiment will be described. While the ACVAE-VC of the above-mentioned conventional method utilizes a training criterion based on speaker identification, this embodiment extends this to a training criterion based on speaker identification. In this embodiment, instead of the training criterion for the speaker identification model expressed by equations (7) and (8), a training criterion for a speaker authentication model based on GE2E loss is introduced.

[0031] The training criterion for the speaker authentication model based on GE2E loss is derived as follows. Assume that the auxiliary classifier consists of a feature extractor and a linear classifier, and the training data contains samples of K classes (K is an integer equal to or greater than 2). As a mini-batch during training, M (M is an integer equal to or greater than 2) training samples {X m , y m} m is given, equation (8) is approximated by the following equation (9).

[0032]

[0033] In formula (9), r m、k is the logit, which is defined by the following equation (10).

[0034]

[0035] In formula (10), w k ∈[w 1 T , ..., w K T ] T (=W) (T is transpose) and b k ∈[b 1 , ..., b K ] (= b) is a learning parameter. m = f ψ (X m) is the intermediate output (speaker embedding representation) of the auxiliary classifier in the speaker verification model.

[0036] Since equation (9) is expressed in the same way as the GE2E loss, it is possible to derive the GE2E loss by modifying the mini-batch creation method and logit as follows. In mini-batch learning, for example, n pieces of training data are extracted from N pieces (N is an integer equal to or greater than 2) and parameters are updated. The extracted training data is the mini-batch.

[0037] Assume that M training samples are composed of I utterances (I is an integer equal to or greater than 2) from J different speakers (J is an integer equal to or greater than 2). In this case, M = IJ. If the mth training sample corresponds to the i-th utterance of the j-th speaker, the index m can be expressed as (i, j). Then, the logit r in Equation (9) m,k (=r i,j,k ) is modified as in the following equation (11).

[0038]

[0039] In equation (11), w and b are learning parameters. k (The following equation (12)) is the centroid containing the i-th utterance. j \i (The following formula (13)) is the centroid excluding the i-th utterance. "\i" indicates that i is excluded.

[0040]

[0041]

[0042] Similarly, the GE2E loss can be derived for equation (7), and the auxiliary classifier can be regarded as a speaker authentication model.

[0043] Furthermore, in this embodiment, since the speaker embedding representation e is used as a condition for the speech conversion model instead of the label y, equation (6) becomes equation (14) below, and equation (7) becomes equation (15) below. That is, in this embodiment, equation (9) is used, which introduces a training criterion for a speaker authentication model based on GE2E loss, instead of equation (8) for the cross-entropy in ACVAE-VC. Also, in this embodiment, equations (14) and (15) are used as training criteria for the speech conversion model and the speaker authentication model, instead of equations (6) and (7) for the variational lower bound in ACVAE-VC. Note that f included in equations (14) and (15) ψ (•) is the output of the speaker authentication model 24 .

[0044]

[0045]

[0046] Equations (14) and (15) are the training criteria for the speech conversion model 23 and the speaker authentication model 24. Therefore, the speaker embedding representations are trained taking into account the speech conversion task. Furthermore, in this embodiment, regularization based on the cosine similarity of the following equation (16) is performed so that the speaker embedding representations before and after the speech conversion model 23 are more consistent.

[0047]

[0048] In addition, in equation (16), cos(·) on the right side is regularization based on cosine similarity. a is the speech signal of the input speaker, and X b is the speech signal of the target speaker. ψ (·) is the output of the speaker authentication model 24. During inference, the acoustic feature sequence X of the speech is converted into the embedding representations e s , e t is used to convert it into the following equation (17):

[0049]

[0050] In addition, in the formula (17), μ φ (·) is a parameter that represents the distribution of the encoder. μ θ(•) is a parameter that represents the distribution of the composite.

[0051] In particular, in the case of one-shot speech conversion, where speech samples from an input speaker and a target speaker are given one utterance at a time, the acoustic feature sequence X is obtained using a speaker authentication model. s , X t From the embedding representation e s (=f ψ (X s )), e t (=f ψ (X t ) can be extracted. The converted acoustic feature sequence is then synthesized using, for example, a neural vocoder. One-shot speech conversion is a technique in which, given two speech samples, one utterance from each speaker, is converted to make the utterance of one speaker sound as if it were being spoken by the other speaker.

[0052] 1 is a diagram showing an example of the configuration of a voice conversion system according to this embodiment. The voice conversion system 1 includes a learning device 2 and a voice conversion device 3.

[0053] The learning device 2 includes, for example, an acquisition unit 21, a learning unit 22, a voice conversion model 23, and a speaker authentication model 24. The learning device 2 may also include a storage unit 25. The voice conversion model 23 includes, for example, an encoder 231 and a decoder 232. The speaker authentication model 24 includes, for example, an auxiliary classifier 241. The auxiliary classifier 241 includes, for example, a feature extractor 242 and a linear classifier 243.

[0054] The voice conversion device 3 includes, for example, an acquisition unit 31, a conversion unit 32, and an output unit 33. The conversion unit 32 includes, for example, an extraction unit 321 and a synthesis unit 322. The voice conversion device 3 may include a storage unit 34.

[0055] The learning device 2 and the voice conversion device 3 are connected to each other by wire or wirelessly. Note that the configuration shown in FIG. 1 is an example and is not limited to this. For example, the voice conversion device 3 may be equipped with the learning device 2. Furthermore, the acquisition unit 21 of the learning device 2 and the acquisition unit 31 of the voice conversion device 3 may be shared. Furthermore, the memory unit 25 and the memory unit 34 may be shared, and the memory unit may be located on the cloud or connected via a network.

[0056] The learning device 2 uses the acquired speech signal by the above-described learning method to learn a voice conversion model 23 based on speaker authentication. The learning device 2 also learns the voice conversion model 23 and the speaker authentication model 24 simultaneously.

[0057] The acquisition unit 21 acquires a voice signal. The voice signal is associated with identification information that can identify the speaker. The identification information may be provided by the user of the learning device 2, or may be the result of voice recognition of information in which a name or the like is spoken at the beginning or end of a voice. The acquired voice signal may be an analog signal or a digital signal. If the voice signal is an analog signal, the acquisition unit 21 converts the analog signal into a digital signal.

[0058] The training unit 22 trains a training standard for the voice conversion model based on, for example, speaker authentication. Specifically, the training unit 22 trains the voice conversion model so that the output of the voice conversion model 23 is authenticated by the speaker authentication model 24 as the target speaker. The training unit 22 trains the speaker authentication model 24 simultaneously with the voice conversion model 23. Furthermore, the training unit 22 trains the output f of the speaker authentication model 24 using the training sample to be converted and the training sample of the target voice as described above. ψ (·) is regularized based on cosine similarity using equation (16). Note that learning is performed using speech data from a predetermined number of people, for example.

[0059] The voice conversion model 23 is a model used for voice conversion. The voice conversion model 23 is, for example, a deep neural network (DNN). The encoder 231 is an encoder that encodes a voice signal. The decoder 232 is a decoder that converts the encoded information into the voice of another person.

[0060] The speaker authentication model 24 is a model used for speaker authentication. The speaker authentication model 24 is, for example, a deep neural network (DNN). As will be described later, the output of the speaker authentication model 24 is a speaker embedding representation.

[0061] The storage unit 25 may store, for example, the acquired speech signal in association with speaker identification information. The storage unit 25 may also store, for each speaker, an acoustic feature in association with the acquired speech signal. The storage unit 25 may also store the type of parameters of each model that are updated by learning.

[0062] The voice conversion device 3 converts the voice signal of the speaker into the voice signal of the target person (also called the target voice) using a trained voice conversion model 23 and a trained speaker authentication model 24 .

[0063] The acquisition unit 31 acquires, for example, a voice signal of a speaker and a voice signal of a target person to be converted. Note that, when the voice signal has already been acquired and stored in the storage unit 25, the acquisition unit 31 may acquire identification information of the speaker and identification information of the target person.

[0064] The conversion unit 32 converts the speaker's voice into a target voice using the trained voice conversion model 23 and speaker authentication model 24 .

[0065] The output unit 33 outputs the audio signal converted by the conversion unit 32 to an external device (for example, a speaker or a terminal equipped with a speaker). Note that the output unit 33 may be equipped with a speaker.

[0066] The storage unit 34 stores, for example, a program used by the conversion unit 32 .

[0067] The learning device 2 and the voice conversion device 3 each include a processor such as a CPU (Central Processing Unit) and memory. The learning device 2 functions as an acquisition unit 21 and a learning unit 22 when the processor executes a program. The voice conversion device 3 functions as an acquisition unit 31, a conversion unit 32, and an output unit 33 when the processor executes a program. Note that all or part of the functions of the learning device 2 and the voice conversion device 3 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above programs may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., solid-state drives (SSDs)), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above programs may be transmitted via telecommunications lines.

[0068] (Learning Process) Next, an example of the learning process performed by the learning device 2 will be described. Fig. 2 is a diagram showing an example of the learning process procedure performed by the learning device according to this embodiment. The learning unit 22 acquires, via the acquisition unit 21, a speech signal whose speaker is known, identification information for identifying the input speaker, and identification information for the target speaker (step S1). For example, the input speaker is A and the target speaker is B.

[0069] The learning unit 22 inputs the acquired input signal to the voice conversion model 23, and converts it into a voice signal of the target speaker (step S2).

[0070] The speech signal of the target speaker A converted by the speech conversion model 23 is input to the speaker authentication model 24 to authenticate the speaker (step S3). Note that the information output from the speech conversion model 23 to the speaker authentication model 24 is, for example, the speech signal of the target speaker.

[0071] In this process, the training unit 22 trains the voice conversion model 23 so that the output (speaker-converted voice signal) of the voice conversion model 23 can be authenticated by the speaker authentication model 24 as the target speaker B (step S4). As described above, the training unit 22 uses equations (14) and (15) as training criteria for the voice conversion model 23 and the speaker authentication model 24. In the voice conversion model 23, the training unit 22 performs regularization based on the cosine similarity of equation (16) on the voice signal of the input speaker and the voice signal of the target speaker.

[0072] In the speaker authentication model 24, a feature extractor 242 extracts acoustic features from the input speech signal, and a linear classifier 243 authenticates the speaker based on the extracted acoustic features. In the speech conversion model 23, an encoder 231 encodes the speech signal of the input speaker. Then, a decoder 232 converts the speech signal into the speech signal of the target speaker using the acquired identification information of the target speaker and the encoded information.

[0073] The learning device 2 repeatedly performs the above process on the utterances of a plurality of speakers.

[0074] The above-described learning procedure is an example and is not limiting. For example, the voice conversion model 23 may convert the speech signal of the target speaker into acoustic features. In this case, the speaker authentication model 24 learns to authenticate the speaker using the acoustic features. Furthermore, the conversion unit 32 of the voice conversion device 3 may convert the acoustic features of the speech signal of the target speaker converted by the voice conversion model 23 into a speech signal.

[0075] (Voice Conversion Processing) Next, an example of the voice conversion processing performed by the learning device 2 will be described. Fig. 3 is a diagram showing an example of the voice conversion processing procedure performed by the voice conversion device according to this embodiment. The acquisition unit 31 acquires the voice signal of the input speaker and the identification information of the target speaker (or the voice signal of the target speaker) (step S11).

[0076] The conversion unit 32 authenticates the input speaker using the speaker authentication model 24 for the acquired speech signal of the input speaker (step S12).

[0077] When the speech signal of the target speaker is acquired, the conversion unit 32 authenticates the target speaker using the speaker authentication model 24 for the acquired speech signal of the target speaker (step S13).

[0078] The conversion unit 32 inputs the speech signal of the input speaker and the identification information of the target speaker into the speech conversion model 23, and converts it into a speech signal of the target speaker (step S14).

[0079] The output unit 33 outputs the converted speech signal of the target speaker to an external device (step S15).

[0080] The above-described voice conversion procedure is an example and is not limiting.

[0081] (Evaluation Results) Next, an example of the results of evaluating the speech conversion system 1 of this embodiment will be described. In the evaluation, speech conversion evaluation between arbitrary speakers was performed assuming one-shot speech conversion. The dataset used was English speech data consisting of 18 speakers called CMU Arctic (http: / / festvox.org / cmu_arctic / ). Of the 592 utterances read aloud by all speakers, 512 utterances were used as training data, and 16 utterances were used as evaluation data. One male and one female (rms and slt) were used as unknown speakers only for evaluation, and another male and one female (bdl and clb) were used as known speakers for training and evaluation. The remaining speakers were used only for training. The speech sampling rate was 16 kHz.

[0082] To confirm the effectiveness of this embodiment, the AutoVC method, a conventional technique, was used as a baseline. The evaluation also evaluated speech synthesized by a neural vocoder from the acoustic features of the input speech and the target speech. The evaluation used Mel-cepstrum distortion (MCD) of the converted speech as an objective evaluation index, and a listening experiment was conducted to investigate the degree to which the converted speech resembled the target speech as a subjective evaluation index. The evaluation results indicated that the smaller the MCD, the closer the converted speech was to the target speech, and the larger the subjective evaluation value, the more similar the converted speech was to the target speaker.

[0083] Fig. 4 is a diagram showing an example of an objective evaluation result. Fig. 5 is a diagram showing an example of a subjective evaluation result. In Fig. 4, the vertical axis is MCD [dB] and the horizontal axis is item. In Fig. 5, the left vertical axis is evaluation percentage (%), the right vertical axis is average preference score of evaluated stores, and the horizontal axis is item.

[0084] In Figures 4 and 5, symbols g1 and g11 indicate the evaluation results of speech synthesized by a neural vocoder from the acoustic features of input speech (Source feature). Symbols g2 and g12 indicate the evaluation results of the AutoVC method. Symbols g3 and g13 indicate the evaluation results of the method of this embodiment, and symbols g4 and g14 indicate the evaluation results of speech synthesized by a neural vocoder from the acoustic features of target speech (Target feature). Note that in Figures 4 and 5, the method of this embodiment is represented as "JSV-VC."

[0085] In FIG. 5, the symbol g21 represents "different", the symbol g22 represents "slightly different", the symbol g23 represents "slightly similar", and the symbol g24 represents "same".

[0086] Fig. 6 is a diagram showing an example of an acoustic feature sequence of a speech signal of an input speaker. Fig. 7 is a diagram showing an example of an acoustic feature sequence of a speech signal converted by the AutoVC method, which is a conventional technique. Fig. 8 is a diagram showing an example of an acoustic feature sequence of a speech signal of a target speaker. Fig. 9 is a diagram showing an example of an acoustic feature sequence of a speech signal converted by the method of this embodiment. In Figs. 6 to 9, the horizontal axis represents time (seconds) and the vertical axis represents Mel frequency bins.

[0087] As shown in Figures 4 and 5, the objective and subjective evaluation results confirmed that the method of this embodiment achieves higher quality conversion performance than the conventional technology. Furthermore, as shown in Figures 6 to 9, it was confirmed that the method of this embodiment is more similar to the target speech than the conventional technology in terms of actual feature values. These results indicate that the method of this embodiment can achieve high-quality speech conversion both quantitatively and qualitatively.

[0088] Furthermore, in the learning device 2 configured in this manner, the voice conversion model is trained so that the output of the voice conversion model is authenticated as the target speaker by the speaker authentication model, so that the conversion process can be taken into account during learning in the voice conversion model in the same way as during inference.

[0089] Furthermore, in the learning device 2 configured as above, the speaker authentication model is trained simultaneously with the voice conversion model, so that speaker embedding expressions can be extracted taking into account the voice conversion task.

[0090] Furthermore, in the learning device 2 configured as above, regularization based on cosine similarity is performed during learning, resulting in a speaker embedding representation that is more consistent before and after speech conversion.

[0091] Furthermore, the speech conversion system of this embodiment can convert non-linguistic and paralinguistic information while preserving the linguistic information contained in the input speech signal, and can be applied to various tasks such as text-to-speech synthesis, speech recognition, and assisting and assisting with vocalization.

[0092] (Variations) In the above example, the learning device 2 is provided with the voice conversion model 23 and the speaker authentication model 24, but this is not limiting. For example, the voice conversion model 23 and the speaker authentication model 24 may be stored on the cloud, or may be connected to the learning device 2 or the voice conversion device 3 via a network. Alternatively, the voice conversion model 23 and the speaker authentication model 24 may be provided in the voice conversion device 3.

[0093] In the above example, the voice conversion device 3 is described as an example of a device, but the present invention is not limited to this. The voice conversion device 3 may be, for example, an application executed on a mobile terminal or a personal computer. In this case, the application performs voice conversion processing using the voice conversion model 23 and the speaker authentication model 24. Furthermore, this application may be stored on a cloud.

[0094] In the above example, the timing of training is not limited to before speech conversion. For example, the speech conversion system 1 may acquire feedback information on the conversion from the user and update the trained speech conversion model 23 and speaker authentication model 24.

[0095] In the above example, the voice conversion system 1 may add or embed, for example, a predetermined signal or identification information of the input speaker to the converted voice signal, for example, at least once or at predetermined intervals, in order to prevent spoofing. Alternatively, the voice conversion system 1 may add or embed, for example, a predetermined signal indicating that the voice signal is a converted voice signal, for example, at least once or at predetermined intervals. Note that the frequency of the predetermined signal is preferably a frequency outside the band of ordinary voices.

[0096] Furthermore, in the above-described example, there may be one or more voice conversion models 23 and speaker authentication models 24. For example, the voice conversion system 1 may have a voice conversion model 23 and a speaker authentication model 24 for each language. In this case, the voice conversion system 1 may recognize the language corresponding to the voice signal of an input speaker using, for example, a known voice recognition technique, or the language may be set by the user. The voice conversion system 1 may then perform training and voice conversion on the voice conversion model 23 and the speaker authentication model 24 associated with the recognized or set language.

[0097] In the above-described embodiments and evaluations, examples of converting the speech signal of the input speaker have been described, but this is not limiting. For example, if the learning device 2 has already learned the speech signal of the input speaker, the learning device 2 may acquire the identification information of the input speaker and the text information to be converted into speech signals of the target speaker based on the acquired information.

[0098] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0099] The present invention is applicable to voice conversion devices, voice conversion applications, and the like.

[0100] 1...Speech conversion system, 2...Learning device, 3...Speech conversion device, 21...Acquisition unit, 22...Learning unit, 23...Speech conversion model, 24...Speaker authentication model, 25...Storage unit, 31...Acquisition unit, 32...Conversion unit, 33...Output unit, 321...Extraction unit, 322...Synthesis unit, 34...Storage unit

Claims

1. A learning device comprising: a voice conversion model that converts a voice signal into the voice signal of another person; a speaker authentication model that authenticates speakers; and a learning unit that uses the voice signals of multiple speakers and identification information that identifies the speakers to learn a learning criterion for the speaker authentication model based on GE2E (Generalized End-to-End) loss simultaneously with the voice conversion model.

2. The learning device according to claim 1, wherein the learning unit learns the voice conversion model such that an output of the voice conversion model is authenticated as a target speaker by the speaker authentication model.

3. The learning device according to claim 1, wherein the learning unit performs regularization based on cosine similarity in the voice conversion model using a training sample to be converted and a training sample of a target voice.

4. The learning device according to claim 1, wherein the speech conversion model uses speaker embedding representations as a learning criterion condition.

5. A voice conversion system comprising: at least one learning device according to claims 1 to 4; and a voice conversion device that converts a voice signal into a voice signal of another person using the trained voice conversion model and the trained speaker authentication model.

6. A training method for a learning device having a voice conversion model that converts a voice signal into the voice of another person and a speaker authentication model that authenticates a speaker, wherein a training unit uses voice signals of multiple speakers and identification information that identifies the speakers to train a learning criterion for the speaker authentication model based on GE2E (Generalized End-to-End) loss simultaneously with the voice conversion model.

7. A program for causing a computer to function as the learning device according to claim 6.

Citation Information

Patent Citations

  • Voice conversion learning device, voice conversion device, method, and program

    JP2019215500A