Voice synthesizer and voice synthesis method

Abstract

To realize conversation with expressive synthesized voice.SOLUTION: A voice synthesizer comprises: a voice acquisition unit for acquiring voice uttered by a speaker; a voice recognition unit for converting the acquired voice into a phoneme sequence; a voice synthesis unit for converting the phoneme sequence into synthesized voice and outputting it; an image acquisition unit for acquiring an image of the speaker; an image recognition unit for recognizing a lip movement pattern from the acquired image; and a voice correction unit for correcting the synthesized voice by using a filter based on the recognized lip movement pattern.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice synthesis device and a voice synthesis method. [Background technology]

[0002] There are known technologies for synthesizing a speaker's voice. Patent Document 1 discloses a technology for synthesizing an image showing lip movement or sign language with audio information. Patent Document 2 discloses a technology that can realize a natural link between multiple media such as animation and synthesized sound. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-226050 [Patent Document 2] Japanese Patent Application Publication No. 10-320170 Summary of the Invention [Problem to be solved by the invention]

[0004] Recent voice conversion and speech synthesis technologies can read text freely while maintaining the target speaker's voice quality. However, conventional technologies place emphasis on reproducing the speaker's voice quality, and therefore often lack expressiveness because they are unable to express the fluctuations contained in real conversation, such as a lack of intonation or a uniform speaking speed.

[0005] An object of the present invention is to provide a speech synthesis device and a speech synthesis method capable of realizing conversation using expressive synthetic speech. [Means for solving the problem]

[0006] The speech synthesis device of the present invention comprises a speech acquisition unit that acquires speech uttered by a speaker, a speech recognition unit that converts the acquired speech into a phoneme string, a speech synthesis unit that converts the phoneme string into synthetic speech and outputs it, an image acquisition unit that acquires an image of the speaker, an image recognition unit that recognizes a lip movement pattern from the acquired image, and a speech correction unit that corrects the synthetic speech with a filter based on the recognized lip movement pattern.

[0007] The speech synthesis method of the present invention includes the steps of acquiring speech uttered by a speaker, converting the acquired speech into a phoneme string, converting the phoneme string into synthetic speech and outputting it, acquiring an image of the speaker, recognizing a lip movement pattern from the acquired image, and correcting the synthetic speech with a filter based on the recognized lip movement pattern. Effect of the Invention

[0008] According to the present invention, conversation can be realized using expressive synthetic speech. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a voice synthesis system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a phoneme estimation result according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing a first example of the correction table according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a second example of the correction table according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing the flow of the voice synthesis method according to the first embodiment. [Figure 6] FIG. 6 is a block diagram illustrating an example of the configuration of a learning device according to the second embodiment. [Figure 7] FIG. 7 is a block diagram showing an example of the configuration of a voice synthesis system according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing the flow of the voice synthesis method according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated explanations will be omitted.

[0011] [First embodiment] (Speech synthesis system) An example of the configuration of a voice synthesis system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a voice synthesis system according to the first embodiment.

[0012] 1, the voice synthesis system 1 includes a microphone 10, a camera 12, a speaker 14, and a voice synthesizer 16. The voice synthesis system 1 is a system in which the voice synthesizer 16 corrects the voice of a speaker detected by the microphone 10 based on a face image of the speaker captured by the camera 12, and outputs the corrected voice from the speaker 14.

[0013] The microphone 10 detects the speaker's voice. The microphone 10 converts the speaker's voice into a voice signal. The microphone 10 outputs the voice signal to the voice synthesizer 16.

[0014] The camera 12 captures an image of the speaker. The camera 12 captures an image of the speaker's face. The camera 12 captures an image of the speaker's mouth, for example. The camera 12 captures an image of the speaker and generates a moving image. The camera 12 outputs the moving image of the speaker to the voice synthesizer 16. Hereinafter, the moving image may be simply referred to as an image.

[0015] The microphone 10 and the camera 12 may be, for example, a device configured as an integrated unit so that the speaker can manually bring the microphone 10 and the camera 12 close to the face. The microphone 10 and the camera 12 may be, for example, devices configured separately from each other.

[0016] The speaker 14 outputs a voice. The speaker 14 outputs a synthetic voice of the speaker synthesized by the voice synthesizer 16.

[0017] The voice synthesizer 16 generates synthetic voice based on the voice of the speaker detected by the microphone 10. The voice synthesizer 16 corrects the generated synthetic voice based on an image of the mouth of the speaker captured by the camera 12. The voice synthesizer 16 outputs the corrected synthetic voice to the speaker 14.

[0018] (Speech synthesizer) As shown in FIG. 1, the voice synthesizer 16 includes a communication unit 20, a storage unit 22, and a control unit .

[0019] The communication unit 20 is a communication interface that communicates between the voice synthesizer 16 and an external device via a wired or wireless network. When a correction table and a trained model, which will be described later, are stored in an external device such as a server, the communication unit 20 communicates with the external device. The communication unit 20 communicates, for example, between the microphone 10 and the voice synthesizer 16. The communication unit 20 communicates, for example, between the camera 12 and the voice synthesizer 16. The communication unit 20 communicates, for example, between the speaker 14 and the voice synthesizer 16.

[0020] The storage unit 22 stores various types of information. For example, the storage unit 22 stores information such as the contents of calculations and programs of the control unit 24. For example, the storage unit 22 stores a correction table for correcting the speaker's synthetic voice. The correction table will be described later. For example, the storage unit 22 includes at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as a HDD (Hard Disk Drive).

[0021] The control unit 24 controls each unit of the voice synthesizer 16. The control unit 24 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM or a ROM. The control unit 24 executes a program that controls the operation of the voice synthesizer 16. The control unit 24 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 24 may be realized by a combination of hardware and software.

[0022] The control unit 24 includes a voice acquisition unit 30, a voice recognition unit 32, a voice synthesis unit , an image acquisition unit , an image recognition unit , a voice correction unit 40, and an output control unit .

[0023] The voice acquisition unit 30 acquires a voice signal related to the voice uttered by the speaker detected by the microphone 10. The voice acquisition unit 30 performs analog-to-digital modulation on the acquired voice signal. The voice acquisition unit 30 outputs the voice signal subjected to analog-to-digital modulation to the voice recognition unit 32.

[0024] The speech recognition unit 32 executes speech recognition processing on the speech signal received from the speech acquisition unit 30 to recognize phonemes uttered by the speaker. The speech recognition unit 32 recognizes the phonemes and generates a phoneme string including a plurality of phonemes. The speech recognition unit 32 outputs phoneme information related to the recognized phonemes to the speech synthesis unit 34 and the speech correction unit 40.

[0025] Specifically, the voice recognition unit 32 executes a voice recognition process on the voice signal received from the voice acquisition unit 30, and converts the voice into text in sentence units. The voice recognition unit 32 converts the voice into text, for example, using a known voice-to-text conversion technique. The voice recognition unit 32 outputs the result of converting the voice into text. The voice recognition unit 32 may convert the voice into text, for example, using a known acoustic model that takes a voice signal as input and outputs text.

[0026] The acoustic model is a trained model used to extract phonemes. The speech recognition unit 32 estimates the phonemes contained in the input speech using the acoustic model. The speech recognition unit 32 outputs the phoneme estimation results as a table that associates time with phoneme labels.

[0027] 2 is a diagram showing an example of a phoneme estimation result according to the first embodiment. Table TB1 shows the result of associating time with a phoneme label when the speech recognition unit 32 converts a speech "Good morning (ohayogozaimasu)" into text. Table TB1 shows the result from when the phoneme "o" is estimated at time t1 to when the phoneme "u" is estimated at time t14.

[0028] Specifically, examples of the acoustic model include a monophone, a triphone, and a context model. A monophone is an acoustic model that models phonemes independently. A triphone is an acoustic model that models a target phoneme by combining it with three sounds before and after it. A context model is an acoustic model that takes into account the context (e.g., acoustic features).

[0029] The speech recognition unit 32 preferably estimates phonemes by "combinations of three phonemes" extracted using triphones. The speech recognition unit 32 may extract a predetermined number or more of consecutive single phonemes using monophones. The speech recognition unit 32 may recognize contexts such as moras (sound segments having a certain time length), parts of speech, and accent phrases as phonemes using a context model. The speech recognition unit 32 may estimate phonemes using pseudo-phoneme labels obtained by prior clustering using a large-scale data set, for example.

[0030] The speech synthesis unit 34 generates synthetic speech. The speech synthesis unit 34 generates synthetic speech having the characteristics of a specific speaker based on the phoneme information received from the speech recognition unit 32. Specifically, the speech synthesis unit 34 converts the phoneme information received from the speech recognition unit 32 into acoustic features using a speaker voice model having the characteristics of the specific speaker. The speech synthesis unit 34 then generates synthetic speech by passing the voice features through a decoder to generate a voice waveform. Here, the speech synthesis unit 34 does not reflect feature information other than the phoneme information, so the characteristics of the synthetic speech, such as the speaking rate, are constant regardless of the speaker.

[0031] The image acquisition unit 36 ​​acquires a face image including the mouth of a speaker captured by the camera 12. The image acquisition unit 36 ​​performs analog-digital modulation on the acquired image. The image acquisition unit 36 ​​outputs the image subjected to analog-digital modulation to the image recognition unit 38.

[0032] The image recognition unit 38 executes image recognition processing on the face image of the speaker received from the image acquisition unit 36 ​​to recognize the "opening of the mouth" and the "turning of the corners of the mouth". The image recognition unit 38 extracts the feature amounts of the "opening of the mouth" and the "turning of the corners of the mouth" and stores them in the storage unit 22. Specifically, the image recognition unit 38 identifies the position of the mouth from the image of the speaker using landmark detection used in face recognition and the like. The image recognition unit 38 then extracts the change in the position of the mouth over time as the feature amounts of the "opening of the mouth" and the "turning of the corners of the mouth". The image recognition unit 38 associates the feature amounts of the "opening of the mouth" and the feature amounts of the "turning of the corners of the mouth" with the time series of the recognition results of the phonemes from the voice recognition unit 32 and stores them in the storage unit 22.

[0033] The voice correction unit 40 corrects the synthetic voice generated by the voice synthesis unit 34. The voice correction unit 40 modulates the synthetic voice generated by the voice synthesis unit 34 in order to reflect the speech rate and other acoustic features of the original speaker. The voice correction unit 40 corrects, for example, the frequency characteristics of the synthetic voice.

[0034] Specifically, the voice correction unit 40 performs time stretching on the synthetic voice based on a table in which time and phoneme labels are associated, generated by the voice recognition unit 32, in order to express a speaking speed equivalent to that of the voice detected by the microphone 10. The time stretching includes a sample skipping process and a sample interpolation process. After performing time stretching on the synthetic voice, the voice correction unit 40 performs smoothing to prevent noise.

[0035] The voice correction unit 40 converts the smoothed synthetic voice into a frequency domain signal by performing a Short-Time Fourier Transform (STFT) on it. The voice correction unit 40 modulates the frequency domain signal according to the time variation of the feature quantity of "mouth opening" and the feature quantity of "turning of the corners of the mouth".

[0036] Formants change depending on "how open the mouth is" and "how upturned the corners of the mouth are." Formants are frequency components that are emphasized by resonance in the vocal tract. Formants are created by the vocal tract. Formants differ from speaker to speaker. The formant with the lowest frequency is called the first formant. The formant with the next highest frequency after the first formant is called the second formant. In other words, the higher the formant, the higher the frequency.

[0037] In general, when the mouth opens wider, the first formant is emphasized. When the corners of the mouth rise wider, the effective length of the vocal tract is reduced, and higher-order formants are emphasized compared to normal speech. Therefore, the voice correction unit 40 converts the formants in association with the time change of the feature amount of "mouth opening" or the feature amount of "mouth corners rise", thereby making it possible to express smooth changes in voice that follow the actual movement of the speaker's mouth. The feature amount of "mouth opening" and the feature amount of "mouth corners rise" are sometimes collectively referred to as the lip movement pattern. The lip movement pattern is sometimes simply referred to as the lip movement. The voice correction unit 40 corrects the synthetic voice with a filter having a filter characteristic that converts the formant based on the lip movement pattern recognized by the image recognition unit 38. Specifically, the voice correction unit 40 corrects the synthetic voice based on a correction table in which a filter according to the lip movement pattern is associated with each phoneme.

[0038] Fig. 3 illustrates a first example of a correction table according to the first embodiment. Fig. 3 illustrates a first example of a correction table according to the first embodiment.

[0039] 3, the correction table TB2 includes items such as "phoneme" and "movement of corners of the mouth". The correction table TB2 includes information on a plurality of filters corresponding to combinations of "phoneme" and "movement of corners of the mouth". The correction table TB2 is stored in the storage unit 22.

[0040] The term "phoneme" refers to a phoneme of speech. A "phoneme" refers to the smallest unit of speech, such as "a," "i," and "u."

[0041] "Mouth corner movement" indicates the movement pattern of the mouth corners. "Low to low" indicates that the mouth corners moved from a low position to another low position. "Low to high" indicates that the mouth corners moved from a low position to a high position. "High to high" indicates that the lips moved from a high position to another high position. "High to low" indicates that the lips have moved from a high position to a low position. The movement patterns of the corners of the mouth are not limited to these, and may include other patterns.

[0042] In one example shown in Figure 3, when the "phoneme" is "a" and the "movement of the corners of the mouth" is "low to low", the voice correction unit 40 corrects the voice for "a" using "filter a-1".

[0043] The correction table TB2 is a table generated for each speaker who uses the voice synthesis system 1. That is, the filters included in the correction table TB2 are different for each speaker. For example, the filters are generated by previously associating the movement of the corners of the mouth with a correction filter indicating the intonation of the voice. The voice correction unit 40 can express the speaker-likeness including the intonation of the voice from which the synthetic voice is converted by correcting the synthetic voice using such a filter. The correction table TB2 may be a correction table common to multiple speakers. The correction table common to multiple speakers may be, for example, a correction table including a filter indicating the average value of the correspondence between the movement of the corners of the mouth and the intonation of the voice.

[0044] Fig. 4 illustrates a second example of the correction table according to the first embodiment. Fig. 4 illustrates a second example of the correction table according to the first embodiment.

[0045] As shown in Fig. 4, the correction table TB3 includes items such as "phoneme" and "mouth opening". The correction table TB3 includes information on a plurality of filters corresponding to combinations of "phoneme" and "mouth opening". In other words, the correction table TB3 differs from the correction table TB2 shown in Fig. 3 in that "mouth opening" is associated with "phoneme". The correction table TB3 is stored in the storage unit 22.

[0046] "Mouth opening" indicates how open the mouth is. "Small to small" represents a change from a small mouth opening to another small mouth opening. "Small to large" represents a change from a small mouth opening to a large mouth opening. "Large to large" represents a change from a large mouth opening to another large mouth opening. "Large to small" represents a change from a large mouth opening to a small mouth opening.

[0047] As an example shown in Fig. 4, when the "phoneme" is "a" and the "mouth opening" is "small → small", the voice correction unit 40 corrects the voice for "a" using "filter a-1A". By using correction table TB3, the voice correction unit 40 can express the likeness of the original speaker in the synthetic voice, in the same way as when correction table TB2 is used.

[0048] After performing the filter correction, the audio correction unit 40 performs an inverse short-time Fourier transform (iSTFT) on the frequency domain signal to restore it to a time domain signal.

[0049] The output control unit 42 performs digital-to-analog modulation on the signal modulated by the voice correction unit 40. The output control unit 42 controls the speaker 14 to output the synthetic voice transformed by the voice correction unit 40.

[0050] (Voice synthesis method) The voice synthesis method according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of the voice synthesis method according to the first embodiment.

[0051] The voice acquiring unit 30 acquires the voice of the speaker from the microphone 10 (step S10). Then, the process proceeds to step S12.

[0052] The speech recognition unit 32 executes speech recognition processing on the speech acquired by the speech acquisition unit 30 to extract phonemes (step S12). Specifically, the speech recognition unit 32 extracts phonemes from the speech acquired by the speech acquisition unit 30 to generate a phoneme string. Then, the process proceeds to step S14.

[0053] The voice synthesis unit 34 generates synthetic voice based on the phonemes extracted by the voice recognition unit 32 (step S14), and the process then proceeds to step S20.

[0054] The image acquisition unit 36 ​​acquires a face image including the mouth of the speaker from the camera 12 (step S16). Specifically, the image acquisition unit 36 ​​acquires the face image at the same time as acquiring the voice in step S10. That is, the process of step S10 and the process of step S16 are executed in parallel. Then, the process proceeds to step S18.

[0055] The image recognition unit 38 recognizes the movement pattern of the speaker's lips based on the face image acquired by the image acquisition unit 36, and detects the feature amount of the lip movement (step S18). Specifically, the image recognition unit 38 detects the feature amount of the lip movement corresponding to the phoneme extracted in step S12. Then, the process proceeds to step S20.

[0056] The voice correction unit 40 corrects the synthetic voice based on the correction table (step S20). Specifically, the voice correction unit 40 corrects the synthetic voice using a filter according to a combination of a phoneme and lip movement based on the correction table TB1 shown in Fig. 3. Then, the process proceeds to step S22.

[0057] The output control unit 42 controls the speaker 14 to output the corrected synthetic voice (step S22), and then ends the process of FIG.

[0058] As described above, in the first embodiment, the synthetic voice is corrected using a correction table in which phonemes are associated with the movement pattern of the corners of the mouth or the degree of opening of the mouth. In this way, the first embodiment can express the likeness of the speaker from which the synthetic voice is converted. In the first embodiment, the correction table in which the movement pattern of the corners of the mouth or the degree of opening of the mouth is associated is exemplified, but is not limited to this. For example, in the first embodiment, filter correction may be performed using a correction table in which phonemes are associated with other lip movements such as the width or length of the mouth. In the first embodiment, not only filter correction using one correction table but also filter correction using multiple correction tables may be combined.

[0059] [Modification of the first embodiment] A modification of the first embodiment will be described. In the first embodiment, the synthetic voice is corrected using a correction table in which phonemes are associated with the movement patterns of the corners of the mouth or the degree of opening of the mouth, but the present invention is not limited to this. In a modification of the first embodiment, the facial expression of the speaker may be recognized based on the face image of the speaker, and the synthetic voice may be corrected using a correction table in which phonemes are associated with the facial expression.

[0060] Examples of facial expressions include, but are not limited to, eye opening, eyebrow lift, eyeball movement, cheek lift, etc. In a modification of the first embodiment, a correction table may be generated in advance, in which a filter is set according to a combination of a phoneme and eye opening.

[0061] For example, in a modification of the first embodiment, the voice correction unit 40 first corrects the synthetic voice using a correction table in which phonemes are associated with the movement patterns of the corners of the mouth or the degree of opening of the mouth. Next, the voice correction unit 40 corrects the synthetic voice using a correction table in which phonemes are associated with the degree of opening of the eyes. This allows the first embodiment to express the likeness of the speaker from which the synthetic voice is converted.

[0062] In a modification of the first embodiment, the synthetic speech may be corrected using only a correction table in which phonemes and facial expressions are associated. In addition, in a modification of the first embodiment, the filter correction using the correction table in which phonemes and facial expressions are associated may be combined with the filter correction using the correction table in which phonemes and lip movements are associated.

[0063] [Second embodiment] A second embodiment will be described. In the second embodiment, a trained model used when correcting synthetic speech is generated. In the second embodiment, the trained model is used to correct synthetic speech.

[0064] A learning device that generates a trained model according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the configuration of the learning device according to the second embodiment.

[0065] The training device 18 includes a communication unit 50, a storage unit 52, and a control unit 54. The training device 18 generates a trained model for correcting the synthetic voice of a speaker who uses the speech synthesis system. The training device 18 may be configured separately from the speech synthesis device 16, or may be configured integrally therewith.

[0066] The communication unit 50 is a communication interface that performs communication between the learning device 18 and an external device via a wired or wireless network.

[0067] The storage unit 52 stores various types of information. For example, the storage unit 52 stores information such as the contents of calculations and programs of the control unit 54. For example, the storage unit 52 includes at least one of a RAM, a main storage device such as a ROM, and an external storage device such as an HDD.

[0068] The control unit 54 controls each unit of the learning device 18. The control unit 54 has, for example, an information processing device such as a CPU or an MPU, and a storage device such as a RAM or a ROM. The control unit 54 executes a program that controls the operation of the learning device 18. The control unit 54 may be realized by, for example, an integrated circuit such as an ASIC or an FPGA. The control unit 54 may be realized by a combination of hardware and software.

[0069] The control unit 54 includes an image acquisition unit 60 , a feature extraction unit 62 , a voice acquisition unit 64 , a phoneme identification unit 66 , and a learning unit 68 .

[0070] The image acquisition unit 60 acquires an image of the user's face including the mouth via the communication unit 50. Specifically, the image acquisition unit 60 acquires an image including the movement of the user's lips. The image acquisition unit 60 outputs the acquired image to the feature extraction unit 62.

[0071] The feature extraction unit 62 extracts the feature of the user's lip movement based on the image including the user's lip movement acquired by the image acquisition unit 60. Specifically, the feature extraction unit 62 extracts the time changes of the user's "mouth opening" and "turning of the corners of the mouth" as the feature. The feature extraction unit 62 outputs the extracted feature of the user's lip movement to the learning unit 68.

[0072] The voice acquisition unit 64 acquires voice corresponding to the image acquired by the image acquisition unit 60 via the communication unit 50. The voice acquisition unit 64 converts the acquired voice into a voice signal. The voice acquisition unit 64 outputs the voice signal to the phoneme identification unit 66 and the learning unit 68.

[0073] The phoneme identification unit 66 performs a voice recognition process on the voice signal received from the voice acquisition unit 64 to identify the phonemes uttered by the user. The phoneme identification unit 66 outputs the identified phonemes to the learning unit 68.

[0074] The learning unit 68 acquires a voice signal from the voice acquisition unit 64. The learning unit 68 acquires a feature of the user's lip movement from the feature extraction unit 62. The feature of the user's lip movement is specifically a feature of the change in the user's lips over time. The learning unit 68 acquires the phonemes spoken by the user from the phoneme identification unit 66. The learning unit 68 acquires the relationship between the voice signal, the feature of the user's lip movement, and the phonemes as teacher data. The learning unit 68 associates the phonemes at the timing when the user speaks with the lip movement.

[0075] The learning unit 68 executes machine learning based on the acquired teacher data to generate a trained model M1 that outputs a filter for correcting a synthetic voice. The filter output by the trained model M1 may be a filter for converting a formant of a specific frequency, or may be a filter for correcting a wider frequency band. The feature amount of the voice signal and the movement of the lips input to the trained model M1 is, for example, data of a length including phonemes before and after one phoneme of interest. The feature amount of the voice signal and the movement of the lips input to the trained model M1 may be data having a predetermined time length, or may be data of a variable time length that can be set to a time estimated as one phoneme. The feature amount of the voice signal and the movement of the lips is trained in association with the phoneme acquired from the phoneme identification unit 66. The learning unit 68 may use various known methods such as GMM (Gaussian mixture model), GMM-HMM (hidden Markov model using Gaussian mixture model), CNN (convolutional neural network), and RNN (recurrent neural network) as a machine learning method.

[0076] The learning unit 68 may generate the trained model M1 based on, for example, only data of a specific speaker. That is, the learning unit 68 may generate the trained model M1 for each speaker. The learning unit 68 may generate the trained model M1 based on, for example, data of an unspecified number of speakers. That is, the learning unit 68 may generate the trained model M1 common to an unspecified number of speakers.

[0077] A voice synthesis system according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the configuration of the voice synthesis system according to the second embodiment.

[0078] The voice synthesis system 1A includes a microphone 10, a camera 12, a speaker 14, and a voice synthesis device 16A.

[0079] The storage unit 22A of the speech synthesis device 16A stores the trained model M1 created by the learning device 18. The storage unit 22A stores a plurality of trained models M1 generated for each user. The storage unit 22A may store one trained model M1 common to all users.

[0080] The voice recognition unit 32 of the control unit 24A outputs the recognized phonemes to the voice synthesis unit 34 and the voice correction unit 40A.

[0081] The voice correction unit 40A of the control unit 24A corrects the synthetic voice using the trained model M1. Specifically, the voice correction unit 40A inputs the phonemes received from the voice recognition unit 32 and the feature amount of the lip movement received from the image recognition unit 38 to the trained model M1, and estimates a filter for correcting the synthetic voice. The voice correction unit 40A corrects the synthetic voice using the estimated filter.

[0082] (Voice synthesis method) The voice synthesis method according to the second embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the voice synthesis method according to the second embodiment.

[0083] The processes from step S30 to step S38 are the same as the processes from step S10 to step S18 shown in FIG. 5, respectively, and therefore will not be described.

[0084] The voice correction unit 40A inputs the phonemes and the feature amounts of the lip movement into the trained model M1 to estimate a filter for correcting the synthetic voice (step S40), and then proceeds to step S42.

[0085] Speech correction unit 40A corrects the synthetic speech using the estimated filter (step S42), and then proceeds to step S44.

[0086] The process in step S44 is the same as the process in step S22 shown in FIG. 5, and therefore a description thereof will be omitted.

[0087] As described above, in the second embodiment, a filter is estimated by inputting phonemes and lip movement features into a trained model, and the synthetic speech is corrected using the estimated filter. In this way, the second embodiment can express the likeness of the speaker from which the synthetic speech is converted.

[0088] [Modification of the second embodiment] A modified example of the second embodiment will be described. In the second embodiment, a phoneme and a feature amount of lip movement are input to a trained model to estimate a filter, and the synthetic speech is corrected using the estimated filter. However, the present invention is not limited to this. In a modified example of the second embodiment, for example, the synthetic speech may be corrected using a trained model that receives input of a phoneme and a feature amount of a facial expression and outputs a filter for correcting the synthetic speech.

[0089] For example, in the modified example of the second embodiment, the voice correction unit 40A first corrects the synthetic voice using a trained model that receives phonemes and lip movement features as inputs and outputs a filter for correcting the synthetic voice. Next, the voice correction unit 40A corrects the synthetic voice using a trained model that receives phonemes and eye opening features as inputs and outputs a filter for correcting the synthetic voice. This allows the modified example of the second embodiment to express the likeness of the speaker from which the synthetic voice is converted.

[0090] In a modified example of the second embodiment, the synthetic speech may be corrected using only a trained model that receives phonemes and facial expression features as input and outputs a filter for correcting the synthetic speech.

[0091] [Third embodiment] A third embodiment will now be described. In the third embodiment, the synthetic speech may be corrected taking into account the phoneme lengths of the target speaker.

[0092] The speech recognition unit 32 outputs a table TB1 in which the time at which a phoneme is detected is associated with the phoneme, as shown in FIG. 2. In table TB1, for example, the time from time t1 at which the phoneme "o" is detected to time t2 at which the phoneme "h" is detected varies depending on the speaker. In other words, the length of each phoneme varies depending on the speaker. Therefore, in the third embodiment, the speech correction unit 40 corrects the synthetic speech taking into account the length of each phoneme.

[0093] For example, the voice correction unit 40 corrects the length of the phonemes in the synthetic voice in accordance with the phonemes recognized by the voice recognition unit 32. In this case, for example, the correction table TB2 shown in Fig. 3 and the correction table TB3 shown in Fig. 4 contain information for changing the phoneme length of the synthetic voice in accordance with the phoneme. That is, in the third embodiment, the voice correction unit 40 changes the frequency characteristics and the phoneme length of the synthetic voice based on each correction table.

[0094] The voice correction unit 40 may change the length of a phoneme in a synthetic voice, for example, by using a trained model. In the third embodiment, a trained model may be generated in advance that receives phonemes and lip movement features as input and outputs a filter including information for changing the phoneme length in a synthetic voice according to a speaker. In this way, in the third embodiment, the voice correction unit 40 changes the frequency characteristics and phoneme length of a synthetic voice by using the trained model.

[0095] As described above, the third embodiment corrects the synthetic speech frequency characteristics and phoneme lengths, thereby making it possible to express the likeness of the original speaker in the synthetic speech.

[0096] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. This distribution and integration configuration may be performed dynamically.

[0097] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0098] 1. Speech synthesis system 10. Mike 12 Camera 14 Speaker 16,16A Voice synthesizer 18 Learning Device 20,50 Communications Department 22,22A,52 Storage section 24,24A Control section 30,64 Voice acquisition section 32 Voice Recognition Unit 34 Voice synthesis section 36,60 Image acquisition section 38 Image Recognition Unit 40,40A Audio correction section 42 Output control section 62 Feature Extraction Unit 66 Phoneme identification part 68 Learning Department

Claims

1. A voice acquisition unit that acquires a voice uttered by a speaker; a speech recognition unit that converts the acquired speech into a phoneme string; a speech synthesis unit that converts the phoneme string into synthetic speech and outputs the synthetic speech; an image acquisition unit for acquiring an image of the speaker; an image recognition unit that recognizes a pattern of lip movement from the acquired image; a voice correction unit that corrects the synthetic voice with a filter based on the recognized lip movement pattern; A speech synthesis device comprising:

2. The audio correction unit infers the filter using a trained model; The trained model is generated so as to receive a combination of a phoneme uttered by the speaker and a lip movement pattern as an input, and to output the filter that corrects the synthetic speech corresponding to the phoneme. The speech synthesis device according to claim 1 .

3. The trained model is generated for each speaker. The speech synthesis device according to claim 2.

4. The image recognition unit recognizes a facial expression of the speaker from the acquired image, The voice correction unit corrects the synthetic voice with the filter based on the recognized facial expression of the speaker. The speech synthesis device according to claim 1 .

5. acquiring speech from a speaker; converting the captured speech into a string of phonemes; converting the phoneme string into synthetic speech and outputting the synthetic speech; acquiring an image of the speaker; Recognizing a lip movement pattern from the captured images; augmenting the synthetic speech with a filter based on the recognized lip movement pattern; A speech synthesis method comprising:

Images