Sound-quality conversion device, sound-quality conversion method, sound-quality conversion neural network, program, and recording medium
By using encoders, processes and decoders in neural networks, the problem of difficulty in converting into multi-person voices and generating intermediate voices in the prior art is solved, and effective conversion of characteristic sounds is achieved.
Patent Information
- Application Number
- CN202380046575.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-07-29
- Filing Date
- 2023-07-27
- Publication Date
- 2025-02-07
AI Technical Summary
The existing sound quality conversion technology is difficult to effectively convert into multiple people's voices, and it is impossible to generate intermediate sounds between calming and whispering sounds.
A neural network including encoder, process and decoder is used for sound quality conversion. The encoder extracts features from the input sound, the process retains the vocalization mode features and removes the speaker's features, and the decoder converts the processed features into the target sound.
When a characteristic sound is input, the output sound reflecting the characteristic is output, which can be effectively converted into a voice of multiple people, and generate an intermediate sound between calming and whispering sound.
Smart Images
Figure CN119404248A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a sound-quality conversion device, a sound-quality conversion method, a program, and a recording medium. Background Art
[0002] Due to the recent progress of deep learning technology, the quality of voice synthesis has been greatly improved. Non-patent document 1 is a technology that can generate voice based on text and convert voice quality. Non-patent document 2 is a technology that converts the voice of a speaker other than the speaker of the voice used for learning based on the technology of non-patent document 1, and can convert the voice quality of the voice of any speaker.
[0003] Prior art literature
[0004] Non-patent literature
[0005] Non-patent literature 1: Jaehyeon Kim, Jungil Kong, and Juhee Son, "ConditionalVariational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech," Proceedings of the 38th International Conference on Machine Learning, 2021, Vol.139of PMLR, pp.5530-5540
[0006] Non-Patent Literature 2: "[OV2L Evolving Summit] Session 4 "Talk on Transforming VITS to Any-to-Many VCs" presented by kaffelun", Internet <URL: https: / / youtu.be / uRwFHuXw3Qk> Summary of the invention
[0007] Problems to be solved by the invention
[0008] In the previous voice quality conversion, even if the sound of the conversion source is input as a characteristic sound including a whisper, falsetto, angry voice, etc., it is converted into a calm sound (normal sound) of the conversion target sound for learning. If the whisper, falsetto, angry voice, etc. are learned as the voices of a single speaker, they can be converted into characteristic sounds by specifying the whisper, falsetto, angry voice as the sound of the conversion target. However, in the case of converting to the voices of multiple people, it is necessary to prepare all the whispers, falsettos, and angry voices of each person as the learning sound. In addition, there is a problem that it is impossible to convert to an intermediate sound between a calm sound and a whisper.
[0009] The present invention has been made in view of the above, and an object of the present invention is to output a sound reflecting the characteristic when a characteristic sound is input during voice quality conversion.
[0010] Means for solving problems
[0011] A sound quality conversion device according to one embodiment of the present invention comprises: an input unit which inputs sound data of a conversion source and meta-information to be operated upon during sound quality conversion; a conversion unit which converts the sound quality of the sound data of the conversion source into sound data corresponding to the meta-information using a learned neural network, wherein the neural network comprises: an encoder which inputs sound data, extracts features from the sound data and outputs a first latent representation; a process which converts the first latent representation into a second latent representation which retains a prescribed feature contained in the sound data and removes the feature corresponding to the meta-information, and adds features corresponding to the meta-information of a conversion target to the second latent representation and reversely converts it into a third latent representation; and a decoder which inputs the third latent representation and outputs the sound data of the conversion target.
[0012] A sound quality conversion device according to one embodiment of the present invention comprises: an input unit which inputs sound data of a conversion source and meta-information to be operated upon during sound quality conversion; a conversion unit which uses a learned neural network to convert the sound quality of the sound data of the conversion source into sound data corresponding to the meta-information, wherein the neural network comprises: a second encoder which inputs sound data and outputs a second latent representation which retains a prescribed feature contained in the sound data and removes a feature corresponding to the meta-information; a process which adds a feature corresponding to the meta-information of a conversion target to the second latent representation and reversely converts it into a third latent representation; and a decoder which inputs the third latent representation and outputs the sound data of the conversion target.
[0013] Effects of the Invention
[0014] According to the present invention, when a characteristic sound is input during voice quality conversion, a sound reflecting the characteristic can be output. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 1 is a diagram showing an example of the structure of a sound-quality conversion device according to the present embodiment.
[0016] Figure 2 This is a diagram showing an example of the structure of the neural network according to the first embodiment.
[0017] Figure 3 This is a flowchart showing an example of the flow of processing during voice quality conversion according to the first embodiment.
[0018] Figure 4 This is a diagram showing an example of the structure of a neural network according to the second embodiment.
[0019] Figure 5 This is a flowchart showing an example of the flow of processing during voice quality conversion according to the second embodiment.
[0020] Figure 6 This is a diagram showing an example of a neural network learning method according to the second embodiment. DETAILED DESCRIPTION
[0021] [First embodiment]
[0022] Reference Figure 1 , an example of the structure of the voice quality conversion device 1 of the first embodiment is described. The voice quality conversion device 1 shown in the figure includes: an input unit 11, a conversion unit 12, and a learning unit 13. Each unit of the voice quality conversion device 1 can also be composed of at least one computer having a processing unit, a storage device, etc., and the processing of each unit is executed by a program. The program is stored in the storage device of the voice quality conversion device 1, and can be recorded in a recording medium such as a magnetic disk, an optical disk, a semiconductor memory, or can also be provided through a network.
[0023] The input unit 11 inputs voice data (hereinafter referred to as voice) and speaker information. Specifically, during learning, the input unit 11 inputs the learning voice of the speaker to be able to be converted to each other and the speaker information of the voice. The voice quality conversion device 1 of the first embodiment learns the voice of the speaker to be able to be converted to each other, and can perform voice quality conversion on the learned speakers in a many-to-many manner. Speaker information is an identifier of the speaker. Speaker information is assigned to each speaker of the learning voice. During learning, the learning voice input by the input unit 11 and the speaker information of the learning voice are sent to the learning unit 13. On the other hand, during inference (voice quality conversion), the input unit 11 inputs the voice of the conversion source, the speaker information of the conversion source, and the speaker information of the conversion target. During inference, the voice of the conversion source and the speaker information input by the input unit 11 are sent to the conversion unit 12.
[0024] The conversion unit 12 inputs the sound of the conversion source, the speaker information of the conversion source, and the speaker information of the conversion target into the learned neural network, reflects the way of utterance of the sound of the conversion source, and converts the sound quality of the sound of the conversion source into a sound corresponding to the speaker information of the conversion target. The way of utterance is whispering, false voice, angry voice, etc. For example, when the sound of the conversion source is a whispering voice, the sound of the conversion target is also generated by whispering. The speaker of the conversion source and the speaker of the conversion target are both speakers of the sound used for learning.
[0025] The neural network of this embodiment has: an encoder that extracts features from the sound of the conversion source and outputs a potential expression of the sound of the conversion source; a process that converts the potential expression of the sound of the conversion source into a speaker-independent potential expression that retains the way of utterance and removes speaker characteristics (speaker characteristics), adds the speaker characteristics of the speaker of the conversion target to the speaker-independent potential expression and reversely converts it into the potential expression of the sound of the conversion target; a decoder (vocoder) that inputs the potential expression of the sound of the conversion target and outputs the sound of the conversion target. The details of the model will be described later.
[0026] The learning unit 13 inputs the learning sound, the speaker information of the learning sound, the text of the learning sound, and the information on the way of utterance of the learning sound (hereinafter referred to as the environment), and restricts the distribution followed by the intermediate representation of the variational autoencoder composed of the encoder and the vocoder to the distribution made according to the text and the environment to learn the neural network. In other words, the learning unit 13 learns the neural network so that the vocoder can restore the latent representation output by the encoder to the original sound, and the latent representation that retains the characteristics of the utterance and removes the speaker characteristics is close to the representation made according to the text and the information on the way of utterance that does not contain the characteristics of the speaker. The text is the phonetic information of the learning sound. The environment is, for example, a flag of 0 and 1 indicating the way of utterance of the learning sound such as whispering, false voice, angry voice, etc. When the learning sound is a whisper, the information indicating the whisper is input to the learning unit 13 as the environment. The parameters (neural network) learned by the learning unit 13 are stored in the storage device of the voice quality conversion device 1.
[0027] (Models and Learning)
[0028] Reference Figure 2 , an example of a neural network and an example of learning according to the first embodiment will be described. The neural network 100 shown in the figure includes an encoder 110 , a flow 120 , a vocoder 130 , and a text encoder 140 .
[0029] The structure composed of the encoder 110 and the vocoder 130 is equivalent to a variational autoencoder. When the sound is input to the encoder 110, a latent representation can be obtained. When the latent representation is input to the vocoder 130, the sound is output. The latent representation has information related to the sound.
[0030] When the process 120 inputs the potential expression and the speaker information, it outputs the speaker-independent potential expression that removes the speaker characteristics from the potential expression as much as possible. In addition, the process 120 is a reversible neural network. In the process 120, when the speaker-independent potential expression is input from the reverse direction and the speaker information of the conversion target is added, the potential expression of the conversion target speaker can be obtained. By inputting the potential expression output by the process 120 to the vocoder 130, the voice of the conversion target speaker can be output.
[0031] The text encoder 140 is a neural network used during learning and is not necessary during inference. The text encoder 140 inputs the text and environment of the learning sound and outputs a latent representation that gives the text the environment. The latent representation output by the text encoder 140 is a representation produced based on the text and environment that does not depend on the speaker and does not include speaker characteristics.
[0032] During learning, the learning sound and the speaker information of the learning sound are input to the encoder 110, and the text and environment of the learning sound are input to the text encoder 140. The neural network is learned in such a way that the sound input to the encoder 110 is the same as the sound output from the vocoder 130, and the neural network 100 is learned in such a way that the latent expression output from the process 120 is close to the expression produced based on the information that does not depend on the speaker and output from the text encoder 140. The latent expression output by the encoder 110 is converted and inversely converted through the process 120 and then input to the vocoder 130. The speaker characteristics are removed during the conversion through the process 120, and the speaker characteristics are given during the inverse conversion. The speaker characteristics given during learning and inverse conversion are the speaker characteristics of the learning sound. Learning is performed in such a way that the spectrum of the sound input to the encoder 110 is consistent with the spectrum of the sound output from the vocoder 130. The learning to make the latent expression output from the process 120 close to the expression output from the text encoder 140 is the same as non-patent document 1, and Monotonic Alignment Search can be used. The horizontal axis of the potential expression output by process 120 is time, and the horizontal axis of the expression produced by the text encoder 140 based on the information not depending on the speaker is phoneme. The correspondence between them is obtained by monotonic alignment, and constraints are imposed to make the correspondence closer. In this embodiment, in addition to the phonemes, the environment is also input as the second information to the text encoder 140. Thus, the neural network 100 is learned so that process 120 outputs the potential expression that removes the speaker characteristics and includes the characteristics of the utterance method.
[0033] The learning voices are prepared as voices of people whose voices are to be converted in a many-to-many manner. For example, when the voices of three people, A, B, and C, are learned as learning voices, after learning, the voice quality of A's voice can be converted into the voice of B or C, the voice quality of B's voice can be converted into the voice of A or C, and the voice quality of C's voice can be converted into the voice of A or B.
[0034] When learning, all the learning sounds of the corresponding environment are not necessarily required. Specifically, when the voice quality conversion device 1 corresponds to whispering sound, even if there is no learning sound of C's whispering sound, it is sufficient to have a learning sound of A or B's whispering sound. That is, it is not necessary to prepare all the learning sounds of the environment corresponding to the voice quality conversion device 1 of all speakers to be learned.
[0035] When the learning sound is a sound including the method of utterance, the method of utterance information is also inputted into the text encoder 140 simultaneously with the text. For example, when the sound of A's whisper is learned as the learning sound, the learning sound of A and the speaker information indicating A are inputted into the encoder 110, and the text of the learning sound and the flag indicating the whisper are inputted into the text encoder 140.
[0036] The speaker information input to the neural network 100 can also be said to be meta-information that is desired to be manipulated during voice quality conversion. As described above, when it is desired to control speaker characteristics, the speaker information is input as meta-information. When pitch and intonation are used as speaker information, it is possible to control the pitch and intonation to perform voice quality conversion. By specifying the pitch and intonation, it is possible to output a higher voice, a lower voice, and a voice with controlled intonation of the speaker of the conversion target. On the other hand, the text and environment input to the text encoder 140 are information that remain unchanged during conversion. In other words, they are features contained in the voice that are desired to be retained even after conversion.
[0037] When the intonation is input as the environment to the text encoder 140 together with the text, that is, when the intonation is processed as the invariant information during the voice-quality conversion, the phonemes obtained from the text each have a time length, but the environment has no time length. Therefore, the time length of the phonological information can be made consistent with the time length of the environment in the monotone alignment. For example, the information of the intonation is extracted from the learning sound, and the time length of the intonation is made consistent with the time length of the sound information.
[0038] Furthermore, in order to take into account differences in the learning sound caused by the microphone, space, and other environments, the learning sound with noise added may be input to the encoder 110 and the learning may be performed so that a clean sound is output from the vocoder 130 .
[0039] (Sound quality conversion processing)
[0040] Reference Figure 3 The flow of processing during voice quality conversion will be described.
[0041] In step S11, the input unit 11 inputs the sound of the conversion source, the speaker information of the conversion source, and the speaker information of the conversion target, and sends them to the conversion unit 12. The voice quality conversion device 1 processes the sound in units of a predetermined number of samples (slices). When the sound of the conversion source is input in real time, it can be processed in units of slices in real time and the voice quality conversion can be performed in real time. The speaker of the conversion source and the speaker of the conversion target are any of the speakers of the learning sound.
[0042] In step S12, the conversion unit 12 inputs the sound of the conversion source and the speaker information of the conversion source to the encoder 110, and obtains the potential expression S1 from the encoder 110. The potential expression S1 is a potential expression including the speaker characteristics of the sound of the conversion source.
[0043] In step S13, the conversion unit 12 inputs the latent expression S1 and the speaker information of the conversion source into the process 120 to obtain the speaker-independent latent expression. The speaker-independent latent expression includes the characteristics of the utterance mode of the sound of the conversion source.
[0044] In step S14, the conversion unit 12 adds speaker information of the conversion target, performs inverse conversion on the speaker-independent latent expression through the process 120, and obtains the latent expression S2 of the sound of the conversion target.
[0045] In step S15 , the conversion unit 12 inputs the potential expression S2 and the speaker information of the conversion target to the vocoder 130 , and outputs the speech of the conversion target reflecting the utterance form of the conversion source speech.
[0046] As described above, the voice quality conversion device 1 of this embodiment includes: an input unit 11 that inputs a sound of a conversion source, speaker information of the conversion source, and speaker information of a conversion target; a conversion unit 12 that converts the sound quality of the conversion source sound into a sound corresponding to the speaker information of the conversion target using the learned neural network 100. The neural network 100 includes: an encoder 110 that inputs a sound, extracts features from the sound, and outputs a potential representation S1; a process 120 that converts the potential representation S1 into a speaker-independent potential representation that retains the features of the utterance method contained in the sound and removes the speaker characteristics of the conversion source, and reversely converts the speaker-independent potential representation into a potential representation S2 by adding the speaker characteristics of the conversion target; and a vocoder (130) that inputs the potential representation S2 and outputs the sound of the conversion target. In this way, the voice quality conversion device 1 can convert the sound quality of the conversion target speaker into a sound that reflects the utterance method of the input sound, such as whispering, false voice, and angry voice. The voice quality conversion device 1 does not specify the pronunciation method of the converted sound, but the encoder 110 and the process 120 output a potential expression of the pronunciation method of the sound of the conversion source. Therefore, for example, when the sound of the conversion source is a sound between a calm sound and a whisper, the output reflects the intermediate pronunciation method.
[0047] The voice quality conversion device 1 of this embodiment includes a learning unit 13 that inputs a learning voice to an encoder 110, and inputs a text of the learning voice and an environment indicating an utterance method included in the learning voice data to a text encoder 140, and learns the neural network 100 so that the vocoder 130 can restore the latent representation output by the encoder 110 to the original learning voice, and the latent representation that is independent of the speaker obtained by the conversion based on the process 120 is close to the representation that is produced based on the information independent of the speaker output by the text encoder 140. Thus, by the conversion in the process 120, the speaker characteristics are removed, and the latent representation including the utterance method is obtained. By inversely converting the latent representation by giving the speaker characteristics of the conversion target speaker to the latent representation, the latent representation including the speaker characteristics and the utterance method of the conversion target speaker can be obtained.
[0048] [Second embodiment]
[0049] The voice quality conversion device of the second embodiment performs additional learning on the neural network 100 of the first embodiment, and performs voice quality conversion on the voice of any speaker. The first embodiment is a voice quality conversion device that performs voice quality conversion in a many-to-many manner. In the second embodiment, after the neural network of the first embodiment is generated, the following learning is performed: a situation in which a potential performance that does not depend on the speaker is obtained without correct speaker information is set as a task. The structure of the voice quality conversion device of the second embodiment is the same as that of the first embodiment, and therefore, the description here is omitted.
[0050] (Models and Learning)
[0051] Reference Figure 4 , an example of a neural network and an example of a learning method of the second embodiment are described. The neural network 100 shown in the figure has: an encoder 110, a process 120, a vocoder 130 and an encoder for any use 150. The encoder 110, the process 120 and the vocoder 130 use the results of the learning completed in the first embodiment. The text encoder 140 is not required when learning in the second embodiment.
[0052] The any encoder 150 is a neural network that inputs a sound without speaker information and outputs a potential expression that does not depend on the speaker. In the second embodiment, the neural network is learned so that the output of the any encoder 150 that inputs the learning sound is close to the potential expression that does not depend on the speaker when there is no speaker information of the learning sound of the conversion source.
[0053] During learning, the learning sound and the speaker information of the learning sound are input to the encoder 110, and the learning sound is input to the any encoder 150. The learning sound used in the first embodiment is also used in the second embodiment. The learning sound is input to the encoder 110 and the any encoder 150, and the neural network is learned so that the potential expression after the output of the encoder 110 is converted in the process 120 is close to the output of the any encoder 150. The potential expression after conversion in the process 120 is a potential expression of the characteristics of the way of utterance that removes the speaker characteristics from the learning sound. The any encoder 150 learns in a way that outputs a potential expression of the characteristics of the way of utterance that removes the speaker characteristics from the input sound. If learning is performed with the learning sounds of multiple speakers ranging from dozens to 100 people, it is considered to be general, and even if the sound of any speaker other than the speaker of the learning sound is input to the any encoder 150, the potential expression of the characteristics of the way of utterance that removes the speaker characteristics can be obtained.
[0054] By inversely transforming the latent representation output by the any encoder 150 through the process 120 and adding the speaker information of the transformation target, the voice quality of the speech input to the any encoder 150 can be transformed into the speech of the speaker of the transformation target.
[0055] (Sound quality conversion processing)
[0056] Reference Figure 5 , the flow of the processing during voice quality conversion in the second embodiment is described.
[0057] In step S21, the input unit 11 inputs the source voice of the conversion and the speaker information of the conversion target, and transmits them to the conversion unit 12. The speaker of the source voice of the conversion may not be the speaker of the learning voice. That is, the voice of any speaker may be input.
[0058] In step S22, the conversion unit 12 inputs the sound of the conversion source to the encoder 150 for any, and obtains the potential expression that is independent of the speaker from the encoder 150. The potential expression that is independent of the speaker includes the characteristics of the utterance form of the sound of the conversion source.
[0059] In step S23, the conversion unit 12 adds speaker information of the conversion target, performs inverse conversion on the speaker-independent latent expression through process 120, and obtains the latent expression S2 of the sound of the conversion target.
[0060] In step S24, the conversion unit 12 inputs the potential expression S2 and the speaker information of the conversion target to the vocoder 130, and outputs the conversion target voice reflecting the utterance form of the conversion source voice.
[0061] (Other learning examples)
[0062] Reference Figure 6 , an example of another learning method of the neural network of the second embodiment is described. Figure 6 The structure of the neural network and Figure 4 The structure of the neural network is the same.
[0063] exist Figure 6 In the learning example of , the neural network is learned in such a way that the potential expression obtained by converting and inversely converting the output of the encoder 110 through the process 120 is close to the potential expression obtained by inversely converting the output of the encoder 150 for any through the process 120. The learning sound is input to the encoder 150 for any. The speaker information of the conversion target is given during the inverse conversion in the process 120. In this way, the learning can also be performed in such a way that the potential expression of the voice of the speaker of the conversion target obtained by the inverse conversion in the process 120 is close.
[0064] Furthermore, the potential representation obtained by the inverse conversion in process 120 may be input to the vocoder 130 to learn the neural network in such a way that the waveforms or spectra are close to each other.
[0065] In addition, Figure 6 In the learning example of , the learning speech and the speaker information S2 of the conversion target may be input to the any encoder 150, and the any encoder 150 may learn by outputting the potential expression S2 without passing through the process 120. In this case, the degree of freedom of the network structure such as the presence or absence of the process 120 can be increased.
[0066] Can also be combined Figure 4 The learning methods shown and Figure 6 The learning method shown.
[0067] As described above, the voice quality conversion device 1 of this embodiment includes: an input unit 11 that inputs a sound of a conversion source and speaker information of a conversion target; a conversion unit 12 that converts the sound quality of the sound of the conversion source into a sound corresponding to the speaker information of the conversion target using the learned neural network 100. The neural network 100 includes: an encoder 150 that inputs a sound and outputs a potential expression that does not depend on the speaker, which retains the characteristics of the utterance method contained in the sound and removes the speaker characteristics of the conversion source; a process 120 that adds the speaker characteristics of the conversion target to the potential expression that does not depend on the speaker and reversely converts it into a potential expression S2; and a vocoder 130 that inputs the potential expression S2 and outputs the sound of the conversion target. As a result, the voice quality conversion device 1 can convert the voice of anyone's voice into the voice quality of the conversion target speaker that reflects the utterance method of the input sound.
[0068] The voice quality conversion device 1 of this embodiment has a learning unit that inputs the learning voice data to the encoder 110 and the any encoder 150 after learning the neural network 100 of the first embodiment, so as to learn the neural network 100 in such a way that the potential expression (supervision) that does not depend on the speaker obtained by the conversion of the process 120 is close to the potential expression output by the any encoder 150. As a result, when the voice of any speaker is input, the any encoder 150 can output the potential expression that removes the speaker characteristics and includes the way of utterance. By inversely converting the potential expression by giving the speaker characteristics of the speaker of the conversion target, the potential expression including the speaker characteristics and the way of utterance of the speaker of the conversion target can be obtained.
[0069] The voice quality conversion device 1 can also learn the neural network 100 in such a way that the potential expression S2 (supervision) obtained by inverse conversion after conversion through process 120 is close to the potential expression S2 obtained by inverse conversion of the speaker-independent potential expression output by the encoder 150 for any through process 120.
[0070] Description of Reference Numerals
[0071] 1 Sound quality conversion device
[0072] 11 Input section
[0073] 12 Conversion Department
[0074] 13 Learning Department
[0075] 100 Neural Networks
[0076] 110 Encoder
[0077] 120 Process
[0078] 130 Vocoder
[0079] 140 Text Encoder
[0080] 150 any encoder.
Claims
1. A sound quality conversion device, characterized in that: have: An input unit that inputs sound data of a conversion source and meta information to be operated upon during sound quality conversion; a conversion unit that converts the sound quality of the sound data of the conversion source into the sound data corresponding to the meta-information using the learned neural network, The neural network has: an encoder that inputs voice data, extracts features from the voice data and outputs a first potential representation; A process of converting the first latent representation into a second latent representation that retains a predetermined feature included in the sound data and removes the feature corresponding to the meta-information, and adding the feature corresponding to the meta-information of the conversion target to the second latent representation to reversely convert the second latent representation into a third latent representation; A decoder inputs the third potential representation and outputs sound data of a conversion target.
2. The sound-to-mass conversion device according to claim 1, characterized in that: The speech quality conversion device comprises: a learning unit, which inputs learning sound data into the encoder, and inputs phonetic information of the learning sound data and an environment representing specified features contained in the learning sound data into the text encoder, so as to enable the decoder to restore the first potential expression output by the encoder to the original learning sound data, and to learn the neural network in a manner such that a second potential expression obtained by converting the first potential expression through the process is close to the expression output by the text encoder.
3. A sound quality conversion device, characterized in that: have: An input unit that inputs sound data of a conversion source and meta information to be operated upon during sound quality conversion; a conversion unit that converts the sound quality of the sound data of the conversion source into the sound data corresponding to the meta-information using the learned neural network, The neural network has: a second encoder which inputs the sound data and outputs a second latent representation which retains the prescribed features contained in the sound data and removes the features corresponding to the meta-information; A process, which adds features corresponding to the meta-information of the conversion target to the second latent representation and reversely converts it into a third latent representation; a decoder, which inputs the third latent representation and outputs the sound data of the conversion target.
4. The sound-to-mass conversion device according to claim 3, characterized in that: The voice quality conversion device includes: a learning unit, which inputs learning sound data into an encoder, and inputs phonetic information of the learning sound data and an environment representing specified features contained in the learning sound data into a text encoder, so that the decoder can restore the first potential expression output by the encoder to the original learning sound data, and learns the neural network in a way that the potential expression obtained by converting the first potential expression through the process is close to the expression output by the text encoder, and then, inputs the learning sound data into the encoder, and inputs the learning sound data into the second encoder, so that the neural network is learned in a way that the potential expression obtained by converting the first potential expression output by the encoder through the process is close to the second potential expression output by the second encoder.
5. The sound-to-mass conversion device according to claim 4, characterized in that: The voice quality conversion device includes: a learning unit that learns the neural network in such a way that a potential expression obtained by converting the first potential expression through the process and then inversely converting it is close to a third potential expression obtained by inversely converting the second potential expression through the process.
6. The sound-mass conversion device according to any one of claims 1 to 5, characterized in that: The meta information is speaker information for identifying a speaker, and the prescribed feature is a way of utterance.
7. A method for converting sound quality, characterized in that: At least one computer inputs the sound data of the conversion source and the meta-information to be operated during the sound quality conversion, and uses the learned neural network to convert the sound quality of the sound data of the conversion source into sound data corresponding to the meta-information. The neural network has: an encoder that inputs voice data, extracts features from the voice data and outputs a first potential representation; A process of converting the first latent representation into a second latent representation that retains a predetermined feature included in the sound data and removes the feature corresponding to the meta-information, and adding the feature corresponding to the meta-information of the conversion target to the second latent representation to reversely convert the second latent representation into a third latent representation; A decoder inputs the third potential representation and outputs sound data of a conversion target.
8. The sound quality conversion method according to claim 7, characterized in that: At least one computer inputs learning sound data into the encoder, and inputs phonetic information of the learning sound data and an environment representing specified features contained in the learning sound data into a text encoder, so that the decoder can learn the neural network in a manner that restores a first potential representation output by the encoder to the original learning sound data, and makes a second potential representation obtained by converting the first potential representation through the process close to the representation output by the text encoder.
9. A method for converting sound quality, characterized in that: At least one computer inputs the sound data of the conversion source and the meta-information to be operated during the sound quality conversion, and uses the learned neural network to convert the sound quality of the sound data of the conversion source into sound data corresponding to the meta-information. The neural network has: a second encoder which inputs the sound data and outputs a second latent representation which retains the prescribed features contained in the sound data and removes the features corresponding to the meta-information; A process, which adds features corresponding to the meta-information of the conversion target to the second latent representation and reversely converts it into a third latent representation; a decoder, which inputs the third latent representation and outputs the sound data of the conversion target.
10. The sound quality conversion method according to claim 9, characterized in that: At least one computer inputs learning sound data into an encoder, and inputs phonological information of the learning sound data and an environment representing specified features contained in the learning sound data into a text encoder, so that the decoder can restore a first potential expression output by the encoder to the original learning sound data, and learn the neural network in a way that the potential expression obtained by converting the first potential expression through the process is close to the expression output by the text encoder. Then, the learning sound data is input to the encoder, and the learning sound data is input to the second encoder, so that the neural network is learned in a way that the potential expression obtained by converting the first potential expression output by the encoder through the process is close to the second potential expression output by the second encoder.
11. The sound quality conversion method according to claim 10, characterized in that: At least one computer learns the neural network in such a way that a potential representation obtained by converting the first potential representation through the process and then inversely converting it is close to a third potential representation obtained by inversely converting the second potential representation through the process.
12. A neural network that inputs voice data and meta-information to be operated at the time of voice-quality conversion and converts the voice-quality conversion into voice data corresponding to the meta-information, characterized in that: The neural network has: an encoder that inputs sound data, extracts features from the sound data and outputs a first latent representation; A process of converting the first latent representation into a second latent representation that retains a predetermined feature included in the sound data and removes the feature corresponding to the meta-information, and adding the feature corresponding to the meta-information of the conversion target to the second latent representation to reversely convert the second latent representation into a third latent representation; A decoder inputs the third potential representation and outputs sound data of a conversion target.
13. A neural network that inputs voice data and meta-information to be operated at the time of voice-quality conversion and converts the voice-quality conversion into voice data corresponding to the meta-information, characterized in that: The neural network has: a second encoder which inputs the sound data and outputs a second latent representation which retains the prescribed features contained in the sound data and removes the features corresponding to the meta-information; A process, which adds features corresponding to the meta-information of the conversion target to the second potential representation and reversely converts it into a third potential representation; The decoder inputs the third potential representation and outputs the sound data of the conversion target.
14. A program, characterized in that A computer is caused to operate as each unit of the sound quality converter according to any one of claims 1 to 5.
15. A recording medium, characterized in that A program for causing a computer to operate as each unit of the sound quality converter according to any one of claims 1 to 5 is recorded.