Learning device, conversion device, learning method, conversion method, and program
The learning device addresses the issue of linguistic information loss in voice conversion by using a mathematical model that integrates audio and lip movement data to convert speech styles, enhancing speech quality and audibility, particularly for whisper-like voices.
Patent Information
- Application Number
- PCT/JP2023/018306
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-21
AI Technical Summary
Conventional voice conversion technologies often lose linguistic information due to noise, especially when converting whisper-like voices that lack pitch information, leading to deterioration in speech quality.
A learning device and method that utilizes a mathematical model incorporating audio data and lip movement video data to convert speech styles, including attribute information, to suppress the deterioration of linguistic information during voice conversion.
The solution effectively preserves linguistic information by using lip video data to enhance the decoding process, resulting in improved speech quality and audibility, even in noisy environments.
Smart Images

Figure JP2023018306_21112024_PF_FP_ABST
Abstract
Description
Learning device, conversion device, learning method, conversion method, and program
[0001] The present invention relates to a learning device, a conversion device, a learning method, a conversion method, and a program.
[0002] In recent years, interest in voice conversion technology has been increasing.
[0003] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo, “ACVAEVC: Non-Parallel Voice Conversion With Auxiliary Classifier Variational Autoencoder,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 9, pp.1432-1443, 2019.Ravindra Yadav, Ashish Sardana, Vinay P Namboodiri, and Rajesh M Hegde, “Speech Prediction in Silent Videos Using Variational Autoencoders,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2021, pp. 7048-7052
[0004] Although various voice conversion technologies have been proposed to date, they are designed to convert sounds that contain pitch information. However, sound is a means of transmitting information that is easily affected by noise. In actual situations where voice conversion technologies are used, the voice often contains noise due to factors such as a noisy surrounding environment. As a result, the voice to be converted often loses pitch information. In such cases, because the pitch information has been lost, linguistic information may be lost during voice conversion.
[0005] Furthermore, speech that does not contain pitch information also includes speech with a voice quality that does not contain pitch information, regardless of the surrounding environment. For example, a whispered voice is one such example. As mentioned above, conventional technology assumes the conversion of sounds that contain pitch information. Therefore, when converting whispered speech, linguistic information may be lost. Furthermore, whispered speech not only does not contain pitch information, but also has a voice quality with low energy. Therefore, it is speech that is easily buried in noise.
[0006] The absence of pitch information means that no comb structure appears in the frequency spectrum. The reason why no comb structure appears in the frequency spectrum of whispered voices is that whispered voices are generated by turbulence and therefore have no fundamental frequency.
[0007] In view of the above circumstances, an object of the present invention is to provide a technique for suppressing the degradation of linguistic information that occurs during speech conversion.
[0008] One aspect of the present invention is a learning device that includes a control unit that learns a mathematical model of a target to be learned, wherein the mathematical model is a mathematical model that converts the target to be learned using voice data of the target to be learned, video data showing the lip movements of a speaker while speaking the voice indicated by the target to be learned, and attribute information that indicates the speaking style of the target to be learned after the conversion.
[0009] One aspect of the present invention is a conversion device comprising: an interface unit that acquires a set of voice data, video data showing the lip movements of a speaker while speaking the voice represented by the voice data, and attribute information that indicates the speaking style of the voice data after conversion; and a control unit that trains a mathematical model of a learning object, wherein the mathematical model is a mathematical model that converts the conversion object using the voice data of the conversion object, video data showing the lip movements of a speaker while speaking the voice represented by the voice data, and attribute information that indicates the speaking style of the conversion object after conversion; and a trained conversion model execution unit that converts the voice data included in the set using the trained mathematical model obtained by the learning device and the set obtained by the interface unit.
[0010] One aspect of the present invention is a learning method including a control step of learning a mathematical model of a learning object, wherein the mathematical model is a mathematical model that converts the conversion object using voice data of the conversion object, video data showing the lip movement of a speaker while uttering the voice indicated by the conversion object, and attribute information that indicates the speaking style of the conversion object after conversion.
[0011] One aspect of the present invention is a conversion method having an interface step of acquiring a set of voice data, video data showing the lip movements of a speaker while uttering the voice represented by the voice data, and attribute information indicating the speaking style of the voice data after conversion, and a control step of training a mathematical model of a training target, wherein the mathematical model is a mathematical model that converts the conversion target using the voice data of the conversion target, video data showing the lip movements of a speaker while uttering the voice represented by the voice data, and attribute information indicating the speaking style of the conversion target after conversion, and a trained conversion model execution step of converting the voice data included in the set using the trained mathematical model obtained by the learning method and the set obtained in the interface step.
[0012] One aspect of the present invention is a program for causing a computer to function as the learning device described above.
[0013] One aspect of the present invention is a program for causing a computer to function as the above-described conversion device.
[0014] The present invention makes it possible to suppress the degradation of linguistic information that occurs during speech conversion.
[0015] FIG. 1 is a diagram showing an example of the configuration of a learning system according to an embodiment. FIG. 2 is an explanatory diagram illustrating an example of a voice data conversion model and learning processing according to an embodiment. FIG. 3 is a diagram showing an example of the hardware configuration of a learning device according to an embodiment. FIG. 4 is a diagram showing an example of the hardware configuration of a conversion device according to an embodiment. A flowchart showing an example of the flow of processing executed by a learning device according to an embodiment. A flowchart showing an example of the flow of processing executed by a conversion device according to an embodiment. A first diagram showing an example of experimental results according to an embodiment. A second diagram showing an example of experimental results according to an embodiment. A third diagram showing an example of experimental results according to an embodiment.
[0016] 1 is a diagram showing an example of the configuration of a learning system 100 according to an embodiment. The learning system 100 includes a learning apparatus 1 and a converting apparatus 2. The learning apparatus 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected via a bus, and executes a program.
[0017] The control unit 11 executes a learning process. The learning process is a process of learning a mathematical model to be learned. The mathematical model to be learned is a mathematical model that converts the audio data to be converted using the audio data to be converted, video data showing the lip movement of a speaker while uttering the audio represented by the audio data to be converted, and attribute information that indicates the speaking style of the audio data after conversion. The audio data is, for example, a frequency spectrogram. Hereinafter, the mathematical model to be learned is referred to as an audio data conversion model.
[0018] The speech data to be converted is, for example, speech data that does not include pitch information, such as whispering.
[0019] Hereinafter, a video showing lip movement will be referred to as a lip video. A lip video is, for example, a video obtained by capturing the face of a speaker while speaking so that the lips are included. A lip video may be, for example, an animation showing the lip movement of a speaker while speaking. Hereinafter, data on a video showing lip movement (i.e., lip video data) will be referred to as lip video data.
[0020] The learning process is executed until a predetermined termination condition for learning (hereinafter referred to as the "learning termination condition") is satisfied. The learning termination condition is, for example, a condition that the change in the learning object due to learning is smaller than a predetermined change. The learning termination condition may also be, for example, a condition that the learning object has been updated a predetermined number of times or more.
[0021] The conversion device 2 converts the input voice data using a trained conversion model, which is the voice data conversion model at the time when the learning termination condition is satisfied. Note that the degree of degradation of linguistic information due to voice conversion depends on the degree of degradation of linguistic information when converting voice data. Therefore, in the learning system 100, in order to suppress the degradation of linguistic information due to voice conversion, the degradation of linguistic information when converting voice data is suppressed.
[0022] 2 is an explanatory diagram illustrating an example of a voice data conversion model and a learning process according to an embodiment. An example of the voice data conversion model and the learning process will be described. In the learning process, a mathematical model to be learned is trained using a set of voice data, lip video data showing the movement of the lips of a speaker while uttering the voice represented by the voice data, and attribute information showing the speaking style of the voice data after conversion.
[0023] Hereinafter, a set of voice data, lip animation data showing the movement of the lips of a speaker while uttering the voice represented by the voice data, and attribute information showing the speech style of the converted voice data used in the learning process will be referred to as learning data. The attribute information is not only used to convert the voice data, but also used during learning to determine whether the speech style of the converted voice data is the speech style indicated by the attribute information.
[0024] The speech style is information indicating the voice quality of the voice indicated by the voice data. Therefore, the speech style indicates, for example, that the voice quality is a whisper. The speech style indicates, for example, that the voice quality is a normal voice. The normal voice is a voice that includes pitch information. The speech style indicates, for example, that the voice quality is a male voice. The speech style indicates, for example, that the voice quality is a female voice.
[0025] The audio data conversion model, which is a mathematical model to be learned, has an audio data encoder, a video data encoder, and an audio data decoder. The audio data encoder encodes the audio data to be processed, and the output follows a normal distribution. In the learning process, the audio data to be processed by the audio data encoder is audio data included in the learning data.
[0026] The video data encoder encodes the video data to be processed, and outputs a video data that follows a normal distribution. The video data is video data. In the learning process, the video data to be processed by the video data encoder is lip video data included in the learning data.
[0027] The audio data decoder is a decoder that decodes audio data encoded by the audio data encoder using the output of the video data encoder and attribute information, and the output follows a normal distribution.
[0028] The normal distribution that the output of the audio data encoder follows is expressed by, for example, the following equation (1).
[0029]
[0030] The normal distribution that the output of the video data encoder follows is expressed by, for example, the following equation (2).
[0031]
[0032] The normal distribution that the output of the audio data decoder follows is expressed by, for example, the following equation (3).
[0033]
[0034] A represents video data. V represents lip video data. y represents the speaking style indicated by the attribute information. Z is a latent variable. More specifically, the latent variable Z represents the output of each encoder. Therefore, the latent variable Z is a quantity that has a strong correlation with the estimated value of the linguistic information.
[0035] μ φA and σ φA 2 Both represent the distribution parameters of the audio data encoder. φVand σ φV 2 Both represent the distribution parameters of the video data encoder. θ (Z, y) and σ θ 2 (Z, y) both represent the distribution parameters of the audio data decoder. A represents the network parameters of the voice data encoder. V represents the network parameters of the video data encoder. θ represents the network parameters of the audio data decoder. φ A and φ V and θ are parameters that are updated through learning.
[0036] As described above, the audio data decoder performs decoding using the output of the video data encoder. The output of the video data encoder may be used in any manner in the audio data decoder as long as the audio data is decoded. For example, the output of the video data encoder may be used in such a manner that the combined output of the video data encoder and the audio data encoder is used as input for the decoding process by the audio data decoder. Note that the combined result means the result of the combination.
[0037] Incidentally, the outputs of the video data encoder and the audio data encoder are specifically matrices, and the integration result may be specifically the sum of the matrix output by the video data encoder and the matrix output by the audio data encoder, or a concatenation in the row direction.
[0038] Therefore, for example, the audio data decoder receives as input the sum of the matrix output by the audio data encoder and the matrix output by the video data encoder.Furthermore, for example, the audio data decoder may receive as input the row-wise concatenation of the matrix output by the audio data encoder and the matrix output by the video data encoder.
[0039] The audio data conversion model performs the process of integrating the output of the video data encoder and the output of the audio data encoder (hereinafter referred to as the "integration process"). That is, the audio data conversion model may include the integration process. The integration process is performed before the audio data decoder executes decoding, and after the audio data encoder and the video data encoder have executed encoding.
[0040] <<Effects of the Integration Result>> The integration result is the sum or concatenation described above. Therefore, when the integration result of the output of the audio data encoder and the output of the video data encoder is used in the decoding process, both the output of the audio data encoder and the output of the video data encoder are explicitly used in the decoding process. When they are used explicitly, the output of the video data encoder has a stronger influence on the decoding process than when they are not used explicitly.
[0041] Furthermore, lip video data is data on video of lips, and lip movement is more resistant to noise than sound, so the stronger the influence of the lip video data, the more the degradation of language information is suppressed. Therefore, degradation of language information is suppressed even more than when it is not explicit. Note that when it is not explicit, it is when the integration result is not used, for example, when it is used in the decoding process as a prior distribution for the output of the audio data encoder. Note that lip movement is more resistant to noise than sound means that language information conveyed by lip movement is resistant to noise because, unlike when it is conveyed by sound, there is no phenomenon of loss of pitch information.
[0042] The learning process involves executing a mathematical model to be learned, a determination process, and a learning control process. The determination process determines whether the voice quality of the speech data output by the mathematical model to be learned is the voice quality indicated by the speaking style. Specifically, the speech data output by the mathematical model to be learned is the speech data output by a speech data decoder.
[0043] The learning control process determines whether the learning termination condition is satisfied. If it is determined that the learning termination condition is not satisfied, the learning control process updates the mathematical model to be learned according to predetermined update rules (hereinafter referred to as "model update rules"). The learning process is executed until the learning termination condition is satisfied.
[0044] <<About Model Update Rules>> An example of the model update rules will be described. As described above, the lip animation data included in the training data is animation data showing the lip movements of a speaker while uttering the sound represented by the sound data included in the training data. Therefore, the linguistic information represented by the lip animation data and the linguistic information represented by the sound data are the same.
[0045] Therefore, in the learning process, the degree of agreement between the normal distribution followed by the output of the audio data encoder and the normal distribution followed by the output of the video data encoder is used as one of the learning criteria, and learning is performed to increase this. In other words, the model update rules include a rule that updates the learning object so as to increase an amount (hereinafter referred to as the "first criterion amount") indicating the degree of agreement between the normal distribution followed by the output of the audio data encoder and the normal distribution followed by the output of the video data encoder.
[0046] The first criterion is, for example, a variational lower bound expressed by the following equation (4): When updating the learning object, the learning object is learned so as to increase the value of equation (4).
[0047]
[0048] Here, the symbols in the following equation (5) represent the sample mean of M pieces of training data (M is 1 or more) used in training the training subject.
[0049]
[0050] The symbol in the following equation (6) represents the Kullback-Leibler (KL) divergence.
[0051]
[0052] Furthermore, the closer the speech style of the speech data output by the speech data decoder is to the speech style indicated by the attribute information, the more preferable it is. Therefore, in the training process, an amount indicating the degree of agreement between the speech style of the speech data output by the speech data decoder and the speech style indicated by the attribute information (hereinafter referred to as the "second criterion amount") may be used as one of the training criteria, and training may be performed to increase this amount. In other words, the model update rule may include a rule that updates the training target so as to increase the second criterion amount.
[0053] The second criterion amount is, for example, the mutual information amount shown in the following equation (7).
[0054]
[0055] I represents the amount of mutual information. The amount indicating the degree of match between the speech style of the speech data output by the speech data decoder and the speech style indicated by the attribute information may be, for example, the variation lower limit indicated by the following equation (8).
[0056]
[0057] Symbol r ψ represents the determination process. t represents the speaking style indicated by the attribute information input to the voice data decoder.
[0058] In the learning process, the determination process r ψ Output and y t The cross-entropy between the training data and the training data (hereinafter referred to as the "third criterion") may be used as one of the training criteria, and training may be performed so as to increase the third criterion. In other words, the model update rule may include a rule that the training target is updated so as to increase the third criterion. Such a cross-entropy is expressed, for example, by the following equation (9).
[0059]
[0060] The symbol in the following equation (10) represents the sample average of M pieces of secondary training data used in training the training target. The secondary training data is a set of audio data and attribute information, excluding lip video data, from the audio data, lip video data, and attribute information included in the training data.
[0061]
[0062] Therefore, the model update rule is, for example, a rule that the learning object is updated so as to increase a linear sum of the first criterion quantity, the second criterion quantity, and the third criterion quantity, the linear sum having a weight of 0 or more. Therefore, the model update rule may be a rule that the learning object is updated so as to increase a linear sum of the first criterion quantity and the second criterion quantity, the linear sum having a weight of 0 or more. Therefore, the model update rule may be a rule that the learning object is updated so as to increase the first criterion quantity. Note that the weight is a predetermined value.
[0063] <About the Conversion Device> As described above, the conversion device 2 converts voice data using a trained conversion model. The conversion device 2 receives as input, as inference stage data, a set of voice data to be converted, lip video data indicating the movement of the lips of a speaker while uttering the voice represented by the voice data, and attribute information indicating the speaking style of the voice data after conversion.
[0064] The voice data included in the inference stage data is, for example, data of voice that does not include pitch information, such as whispering.
[0065] The conversion device 2 converts the voice data including the inference stage data by inputting the inference stage data into the trained conversion model. That is, the conversion device 2 converts the voice data including the inference stage data by executing the trained conversion model with the inference stage data as the execution target of the trained conversion model. Considering how attribute information is used in the trained voice data conversion model, the attribute information included in the inference stage data is information indicating the target voice quality for the voice quality of the converted voice data.
[0066] The processing of the trained conversion model in the conversion device 2 can be expressed by, for example, the following formula (11). In the example of formula (11), the above-mentioned integration processing is executed, and in particular, an example in which a sum is obtained as the integration processing is shown.
[0067]
[0068] y t is attribute information included in the inference stage data. Therefore, it is information input to the voice data conversion model as attribute information in the learning stage of the voice data conversion model. The symbol on the left side of equation (11) represents the output of the learned conversion model. μ θ The first term in the brackets in μ represents the output of the trained speech data encoder. θ The second term in the parentheses represents the output of the trained video data encoder. θ represents the processing of the trained audio data decoder.
[0069] 3 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. As described above, the learning device 1 includes a control unit 11 and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.
[0070] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.
[0071] The control unit 11 performs, for example, a learning process. The control unit 11 controls, for example, the operation of each functional unit included in the learning device 1. The control unit 11 acquires learning data, for example, via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.
[0072] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits the learning data. In such a case, the interface unit 12 acquires the learning data by communicating with the device that transmits the learning data. The external device is, for example, the conversion device 2. The interface unit 12 transmits the trained voice data conversion model (i.e., the trained conversion model) obtained by the training process to the conversion device 2 by communicating with the conversion device 2.
[0073] The interface unit 12 includes input devices such as a mouse, keyboard, and touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the interface unit 12 accepts input of various information to the learning device 1 via the input device, either wired or wireless. Note that the learning data does not necessarily have to be input to a communication interface, but may also be input to an input device.
[0074] The interface unit 12 outputs various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 outputs information that has been input to the interface unit 12, for example.
[0075] The memory unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The memory unit 13 stores various information related to the learning device 1. The memory unit 13 stores, for example, information necessary for the learning process. The memory unit 13 stores, for example, a voice data conversion model in advance. The memory unit 13 stores, for example, various information generated by the operation of the control unit 11. Therefore, the memory unit 13 stores, for example, a trained voice data conversion model. The memory unit 13 stores, for example, information acquired by the interface unit 12.
[0076] 4 is a diagram illustrating an example of a hardware configuration of the conversion device 2 according to an embodiment. As described above, the conversion device 2 includes the control unit 21 (an example of a trained conversion model execution unit) and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.
[0077] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the conversion device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.
[0078] The control unit 21 executes the trained conversion model, for example, with the inference stage data as the execution target. As a result, the control unit 21 converts the voice data included in the inference stage data. That is, the control unit 21 converts the voice data included in the inference stage data using the trained conversion model and the inference stage data.
[0079] The control unit 21 controls, for example, the operation of each functional unit included in the conversion device 2. The control unit 21 acquires, for example, various pieces of information acquired via the interface unit 22. Therefore, for example, the control unit 21 acquires data to be executed on the trained conversion model (i.e., inference stage data). The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.
[0080] The interface unit 22 includes a communication interface for connecting the conversion device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that has transmitted the inference stage data. In such a case, the interface unit 22 acquires the inference stage data by communicating with the device that has transmitted the inference stage data. The external device is, for example, the learning device 1. The interface unit 22 acquires the trained conversion model by communicating with the learning device 1.
[0081] The interface unit 22 is configured to include input devices such as a mouse, keyboard, and touch panel. The interface unit 22 may be configured as an interface that connects these input devices to the conversion device 2. In this way, the interface unit 22 accepts input of various information to the conversion device 2 via the input device, either wired or wireless. Note that the inference stage data does not necessarily have to be input to a communication interface, but may also be input to an input device.
[0082] The interface unit 22 outputs various types of information. The interface unit 22 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the conversion device 2. The interface unit 22 outputs, for example, information input to the interface unit 22.
[0083] The storage unit 23 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the conversion device 2. The storage unit 23 stores, for example, a trained conversion model. The storage unit 23 stores, for example, various information generated by the operation of the control unit 21. The storage unit 23 stores, for example, information acquired by the interface unit 22.
[0084] 5 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The control unit 11 acquires learning data (step S101). The control unit 11 executes a learning process using the acquired learning data until a learning termination condition is met (step S102). The control unit 11 outputs the trained voice data conversion model to a predetermined output destination (step S103). The predetermined output destination is, for example, the storage unit 13. Note that output to the storage unit 13 means writing to the storage unit 13. The predetermined output destination may be, for example, an external device such as the conversion device 2 connected for communication via the interface unit 12.
[0085] 6 is a flowchart showing an example of the flow of processing executed by the conversion device 2 according to the embodiment. In the example of the flowchart in FIG. 6, it is assumed that the trained voice data conversion model has been stored in advance in the storage unit 23.
[0086] Sound data is input to the interface unit 22, and the control unit 21 acquires the inference stage data input to the interface unit 22 (step S201). Inputting the inference stage data to the interface unit 22 means that the interface unit 22 acquires the inference stage data.
[0087] The control unit 21 executes the learned conversion model on the inference stage data obtained in step S201 (step S202). The processing of step S202 obtains speech data after converting the speech data included in the inference stage data obtained in step S201. After step S202, the control unit 21 outputs the speech data obtained in step S202 to a predetermined output destination (step S203). The predetermined output destination is, for example, the storage unit 23. Note that output to the storage unit 23 means writing to the storage unit 23. The predetermined output destination may be, for example, a predetermined external device connected so as to be able to communicate via the interface unit 22.
[0088] <Experiment> An experiment using the learning system 100 and its results will be described. In the experiment, an experiment was conducted to convert the speaking style from whispered to normal speech, assuming a quiet environment and a noisy environment. For the experimental evaluation, a video of a Japanese sentence being read aloud by one Japanese speaker was recorded. The read sentence consisted of a total of 300 utterances, and both normal speech and whispered speech were recorded. Of these, 256 utterances and 16 utterances were used as training data and evaluation data, respectively, in the experiment.
[0089] Mel spectrograms were used as audio data. A series of landmark coordinates near the lips extracted from OpenFace was used as video data. The DEMAND noise database was used to simulate noise environments and for data augmentation.
[0090] In the experiment, a technology that performs voice conversion without using video data was used as a comparison technology and compared with the learning system 100.
[0091] In addition, the experiment also evaluated speech synthesized using a neural vocoder from the acoustic features of the input speech and the target speech. The objective evaluation indices used were the Mel-Cepstral Distortion (MCD) of the converted speech and the Linear Correlation Coefficient (LFC) of the logarithmic F0 sequence.
[0092] In addition, as an experiment to obtain subjective evaluation indices, a listening experiment (MOS test) was conducted to rate the quality and audibility of the converted speech. As an experiment to obtain subjective evaluation indices, a listening experiment (XAB test) was also conducted to evaluate how similar the quality and audibility of the learning system 100 to the target speech were in comparison with a comparative technology.
[0093] The smaller the MCD, the closer the converted speech is to the target speech, and the larger the LFC, the higher the correlation between the pitch information of the converted speech and the target speech. The higher the MOS test score, the higher the quality and ease of listening of the converted speech. The higher the XAB test score, the better the quality and ease of listening of the present invention compared to the prior art.
[0094] 7 to 9 show examples of experimental results of the embodiment. More specifically, FIG. 7 shows the objective evaluation results. In FIGS. 7 to 9, "W2N-SC" indicates the results of the compared technology. "W2N-AVSC" indicates the results of the learning system 100.
[0095] 7 shows that the MCD is comparable between the comparative technology and the training system 100, while the LFC is significantly improved in the training system 100. This indicates that the training system 100 can convert whispered speech into speech with more natural pitch information compared to the comparative technology.
[0096] 8 and 9 show examples of objective evaluation results. More specifically, Fig. 8 shows an example of the results of the MOS test, and Fig. 9 shows an example of the results of the XAB test.
[0097] Figures 8 and 9 show that the conversion results of the comparison technology and the learning system 100 are of similar quality in both quiet and noisy environments, while the learning system 100 provides improved audibility. Figures 8 and 9 also show that in listening experiments using the comparison technology, the learning system 100 is superior to the comparison technology in both quality and audibility. The results of Figures 7 to 9 show that the learning system 100 achieves higher quality speech conversion than the comparison technology.
[0098] The learning device 1 configured in this way also uses lip video data to convert voice data. Lip movements are a means of transmitting linguistic information without using sound, although visual reading requires considerable training. Therefore, even if the voice to be converted is weak and easily affected by ambient noise, the learning device 1 can suppress the degradation of linguistic information that occurs during voice conversion due to the influence of noise.
[0099] Furthermore, the conversion device 2 configured in this manner converts speech data using the learned mathematical model obtained by the learning device 1. Therefore, the conversion device 2 can suppress the degradation of linguistic information that occurs during speech conversion, even if the speech to be converted does not contain pitch information.
[0100] Furthermore, the learning system 100 configured in this manner includes the learning device 1. Therefore, the learning system 100 can suppress the degradation of linguistic information that occurs during speech conversion, even if the speech to be converted does not contain pitch information.
[0101] <Example of application scenario of conversion device 2> The conversion device 2 is, for example, a communication device such as a smartphone. For example, when having a private conversation in a public place, the speaker may whisper to prevent third parties from overhearing the private content or to avoid disturbing others in the public place. However, whispering is difficult to hear not only for third parties but also for the other party on the call.
[0102] In such cases, if the smartphone runs the trained conversion model to convert the input whispered voice data into normal voice data and transmit it to the other party, the other party will hear normal voice rather than whispered voice, which will result in the speaker's linguistic information being more appropriately conveyed to the other party.
[0103] Even if the speaker does not whisper, there may be a lot of ambient noise in public places, and the voice may reach the other party with pitch information lost. Even in such cases, by running a pre-trained conversion model, the linguistic information the speaker wants to convey can be more appropriately conveyed to the other party.
[0104] Furthermore, voice quality is not always consistent during a call. For example, a conversation may contain a mixture of whispered and normal speech. Even in such cases, by running a trained conversion model, the linguistic information that the speaker wants to convey can be more appropriately conveyed to the other party.
[0105] (Modification) The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.
[0106] The conversion device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.
[0107] Note that all or part of the functions of the learning device 1 and the conversion device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.
[0108] The control unit 21 is an example of a trained conversion model execution unit.
[0109] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.
[0110] 100...Learning system, 1...Learning device, 2...Conversion device, 11...Control unit, 12...Interface unit, 13...Memory unit, 21...Control unit, 22...Interface unit, 23...Memory unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory
Claims
1. A learning device comprising: a control unit that learns a mathematical model of a learning target, wherein the mathematical model converts the conversion target using voice data of the conversion target, video data showing the lip movement of a speaker speaking the voice represented by the conversion target, and attribute information that indicates the speaking style of the conversion target after conversion.
2. The learning device of claim 1, wherein the mathematical model comprises: an audio data encoder that encodes audio data to be processed and whose output follows a normal distribution; a video data encoder that encodes video data to be processed and whose output follows a normal distribution; and an audio data decoder that decodes the audio data encoded by the audio data encoder using the output of the video data encoder and the attribute information and whose output follows a normal distribution.
3. The learning device described in claim 2, wherein, in learning the mathematical model, the mathematical model is updated so as to increase a first criterion amount which is an amount indicating the degree of agreement between the normal distribution followed by the output of the audio data encoder and the normal distribution followed by the output of the video data encoder.
4. A conversion device comprising: an interface unit that acquires a set of voice data, video data showing the lip movement of a speaker while speaking the voice represented by the voice data, and attribute information that indicates the speaking style of the converted voice data; and a control unit that trains a mathematical model of the learning target, wherein the mathematical model is a mathematical model that converts the conversion target using the voice data of the conversion target, video data showing the lip movement of the speaker while speaking the voice represented by the conversion target, and attribute information that indicates the speaking style of the conversion target after conversion, and a trained conversion model execution unit that uses the trained mathematical model obtained by the learning device and the set acquired by the interface unit to convert the voice data included in the set.
5. A learning method comprising: a control step of learning a mathematical model of a target to be converted, wherein the mathematical model is a mathematical model that converts the target to be converted using voice data of the target to be converted, video data showing the movement of the lips of a speaker while speaking the voice represented by the target to be converted, and attribute information that shows the speaking style of the target to be converted.
6. A conversion method comprising: an interface step of acquiring a set of voice data, video data showing the lip movement of a speaker speaking the voice represented by the voice data, and attribute information indicating the speaking style of the converted voice data; and a control step of training a mathematical model of a learning target, wherein the mathematical model is a mathematical model that converts the conversion target using the voice data of the conversion target, video data showing the lip movement of a speaker speaking the voice represented by the conversion target, and attribute information indicating the speaking style of the conversion target after conversion, and a trained conversion model execution step of converting the voice data included in the set using the trained mathematical model obtained by the learning method and the set obtained by the interface step.
7. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.
8. A program for causing a computer to function as the conversion device according to claim 4.
Citation Information
Patent Citations
Whisper conversion method, apparatus, device and readable storage medium
JP2020515877A
