Conversion device, conversion method, and conversion program

Abstract

A conversion device (10) comprises: a voice recognition unit (11) that generates, from the voice data of a first person and the voice data of a second person, syllable-unit speech text of the voice data of the first person and of the voice data of the second person; a ratio calculation unit (12) that calculates, for each syllable, the ratio between the speaking time of the first person and the speaking time of the second person; a setting unit (13) that, for each part of a motion generating subject, adjusts the ratio of each syllable and syllable periods; and a conversion unit (14) that generates motion data for the second person by expanding / contracting the motion data of the first person in accordance with the syllable periods of the voice data of the second person.

Description

Conversion device, conversion method, and conversion program

[0001] The present invention relates to a conversion device, a conversion method, and a conversion program.

[0002] A technology has been proposed in which a learning model is created using a set of voice data and motion data acquired by motion capture or the like for a specific person as learning data, and corresponding motion is estimated from the voice data of that person.

[0003] Chaitanya Ahuja1, Dong Won Lee, Yukiko I. Nakano, and Louis-Philippe Morency, “Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach”, arXiv:2007.12553v1, 2020.Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, Louis-Philippe Morency, “No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures”, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1884-1895, 2020.

[0004] However, in the conventional technology, in order to create a learning model, it was necessary to prepare a large number of learning data sets, each set consisting of voice data and motion data of the same person.

[0005] Furthermore, conventional learning models have the problem that they cannot estimate motions from voice data with a different voice tone (different person) than that used during learning, making it impossible to generate motions that match the voice data. In particular, when the person used during learning is a fictitious character created using voice synthesis, it is impossible to obtain motion data for learning, making it impossible to generate motions.

[0006] The present invention has been made in consideration of the above, and aims to provide a conversion device, a conversion method, and a conversion program that can generate motion that matches the voice of a person for whom only voice data exists.

[0007] In order to solve the above-mentioned problems and achieve the object, the conversion device is characterized by having a voice recognition unit that generates syllable-based spoken text of the voice data of a first person and the voice data of a second person from the voice data of the first person and the voice data of the second person, respectively; a calculation unit that calculates the ratio between the speech time of the first person and the speech time of the second person for each syllable; a setting unit that adjusts the ratio for each syllable and the syllable interval for each part of the body that is the target of motion generation; and a conversion unit that stretches or shrinks the motion data of the motion data of the first person to match the syllable interval of the voice data of the second person, thereby generating motion data of the second person.

[0008] According to the present invention, even for a person for whom only voice data exists, motions that match the voice can be generated.

[0009] FIG. 1 is a diagram showing an example of the configuration of a conversion device according to an embodiment. FIG. 2 is a diagram showing an example of input data and output data of a voice recognition unit. FIG. 3 is a diagram showing an example of input data and output data of a ratio calculation unit. FIG. 4 is a diagram explaining processing by a setting unit. FIG. 5 is a diagram explaining processing by the setting unit. FIG. 6 is a diagram explaining processing by a conversion unit. FIG. 7 is a flowchart showing the processing procedure of conversion processing according to an embodiment. FIG. 8 is a diagram explaining an example of application of an embodiment. FIG. 9 is a diagram explaining an example of application of an embodiment. FIG. 10 is a diagram explaining an example of generation of motion data. FIG. 11 is a diagram explaining an example of generation of motion data. FIG. 12 is a diagram explaining an example of generation of motion data. FIG. 13 is a diagram showing an example of a computer on which a conversion device is implemented by executing a program.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail embodiments of a conversion device, a conversion method, and a conversion program according to the present application, with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0011] [Embodiment] In the embodiment, a method is proposed for generating motion data of person X, for which only voice data exists, from data of person A, for which a learning dataset consisting of a combination of voice data and motion data exists.

[0012] In this embodiment, motion data that matches a different voice tone is generated by expanding or contracting the motion data in the time direction for each short section such as a syllable. If the data (voice and motion) of person A is replaced with person A with only voice data, even if the speech content is the same, the speech speed and pauses will be different, and the voice and motion will not match, which is not realistic.

[0013] Therefore, in this embodiment, the required time for each short section such as a syllable is measured, and the motion of person A (first person) in the same section is expanded or contracted according to the required time for person X (second person). In this way, in this embodiment, expanded or contracted motion data of person X, for which only audio data exists, is obtained.

[0014] [Conversion Device] Next, a conversion device according to an embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a conversion device according to an embodiment.

[0015] The conversion device 10 according to the embodiment is a server device that is realized by loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The conversion device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0016] As shown in Fig. 1, the conversion device 10 has a voice recognition unit 11, a ratio calculation unit 12, a setting unit 13, and a conversion unit 14. The conversion device 10 receives as input pair data of voice data of person A and motion data of person A, voice data of person X for which motion data is to be created, and a threshold range of the expansion / contraction ratio set for each body part. The voice data of person X has the same lines as the voice data of person A. The conversion device 10 then outputs the motion data of person X.

[0017] The speech recognition unit 11 generates syllable-based spoken text from the speech data of person A and the speech data of person X. The speech recognition unit 11 generates syllables and the start and end times of the syllables from the input speech data.

[0018] The voice data of person A and the voice data of person X are input to the voice recognition unit 11. The voice recognition unit 11 outputs the syllable-by-syllable spoken text of person A and person X and the speaking times of person A and person X (start time and end time of each syllable) to the ratio calculation unit 12.

[0019] 2 is a diagram showing an example of input data and output data of the speech recognition unit 11. For example, as shown in Fig. 2, when speech data "I understand. Now, please look at the screen" is input to the speech recognition unit 11, the speech recognition unit 11 outputs syllable-based spoken text "I understand," "I did," "I did," "," "Well then," "Screen," "Of," "How," "O," "," "Please," "Please," and "Please" in association with each syllable, along with the start time and end time of each syllable.

[0020] The ratio calculation unit 12 compares the speech times of person A and person X for each syllable and calculates the ratio. The ratio calculation unit 12 receives as input the speech text of person A in syllable units, the speech text of person X in syllable units, the speech time of person A (start time and end time of each syllable), and the speech time of person X (start time and end time of each syllable). The ratio calculation unit 12 calculates the ratio (expansion / contraction ratio) between the speech time of person A and the speech time of person X for each syllable and outputs the ratio to the setting unit 13. The speech time ratio is the value obtained by dividing the speech time of person X by the speech time of person A for the same syllable.

[0021] 3 is a diagram showing an example of input data and output data of the ratio calculation unit 12. Table T1 in Fig. 3 is a syllable-based spoken text of person A. Table T2 is a syllable-based spoken text of person X. The ratio calculation unit 12 calculates the ratio of the speaking time between person A and person X for each syllable.

[0022] For example, consider the syllable "shouchi" ("shouchi"). In this case, the start time of person A's speech is "0.104635" and the end time is "0.39498866", so the speech time of person A is "0.29035366". Also, the start time of person X's speech is "0.0033245" and the end time is "0.235607428", so the speech time of person X is "0.232282928". Therefore, the ratio calculation unit 12 calculates the ratio of the syllable "shouchi" to be "0.8" by dividing the speech time of person X, "0.232282928", by the speech time of person A, "0.29035366" (Table T3).

[0023] The ratio calculation unit 12 calculates the ratio by dividing the speech time of person X by the speech time of person A for each syllable, and outputs the ratio of each syllable (table T3).

[0024] The setting unit 13 receives as input the threshold range of the expansion / contraction ratio for each body part for which motion is to be generated, the ratio for each syllable calculated by the ratio calculation unit 12, and the speech times of person A and person X (start time and end time for each syllable). The setting unit 13 adjusts the ratio for each syllable and the syllable section for each body part for which motion is to be generated. The setting unit 13 adjusts the ratio for each syllable and the syllable section based on the threshold range of the expansion / contraction ratio set for each body part for which motion is to be generated.

[0025] The setting unit 13 adjusts the ratio of each syllable and the syllable section so that the ratio of each syllable falls within the threshold range of the expansion / contraction ratio set for each body part for which motion is to be generated. In other words, the setting unit 13 expands or contracts the syllable section so that the ratio falls within the threshold range of the expansion / contraction ratio set for each body part for which motion is to be generated. The setting unit 13 outputs the adjusted syllable section to the conversion unit 14 as an adjusted expansion / contraction section. The setting unit 13 outputs the adjusted expansion / contraction section for each body part.

[0026] Here, the threshold range of the expansion ratio will be described. The threshold range of the expansion ratio is set for each body part. The threshold range of the expansion ratio is set as the range of the allowable expansion ratio for each body part, such as the face, mouth, body, and arms, and the lower and upper limits of the allowable expansion ratio are set as the threshold range of the expansion ratio. The threshold range of the expansion ratio is adjusted as appropriate, for example, while checking the motion of person X after conversion by the conversion device 10. Furthermore, since there are the following trends for each body part, the threshold range of the expansion ratio is set in accordance with these trends.

[0027] Specifically, for the face (especially around the mouth), a wider threshold range for the expansion ratio and resilience tends to reduce the sense of incongruity in the motion, so the threshold range for the expansion ratio is wider for the face than for other parts. For this reason, for example, the threshold range for the expansion ratio for the face is set to greater than 0.5 and less than 2.0.

[0028] Furthermore, since the threshold range of the expansion ratio for the body (especially the arms) is set narrower and excessively agile or slow movements are avoided to create less discomfort, the threshold range of the expansion ratio is narrower than for other parts of the body. For example, the threshold range of the expansion ratio for the body is set to greater than 0.75 and less than 1.33.

[0029] The setting unit 13 refers to the threshold range of the expansion / contraction ratio for each part, and for a syllable whose ratio calculated by the ratio calculation unit 12 exceeds the threshold range of the expansion / contraction ratio, sets a section in which the following syllables are merged until the ratio falls within the threshold range of the expansion / contraction ratio.

[0030] 4 and 5 are diagrams illustrating the processing performed by the setting unit 13. A case where syllables are adjusted for the face will be described with reference to Fig. 4. A case where syllables are adjusted for the body will be described with reference to Fig. 4.

[0031] For example, as shown in tables T11 and T12 in FIG. 4, the ratio of the syllable " " (cell C14) is "0.5" (cell R14), which is below the threshold range of the facial expansion / contraction ratio (greater than 0.5 and less than 2.0).

[0032] Therefore, the setting unit 13 sets a section (cell C14' in table T13) in which the syllable "Soredemo wa" (cell C15) following the syllable " " is merged with the voice " ". When cell C14' is set, the start time of person A's utterance is "0.9949886" and the end time is "2.0299773", so the utterance time of person A is "1.0349887" (rows L4-1 and L5-1 in table T1 in FIG. 3). When cell C14' is set, the start time of person X's utterance is "1.090607347" and the end time is "1.661600557", so the utterance time of person X is "0.57099321" (rows L4-2 and L5-2 in table T1 in FIG. 3). As a result, the ratio of cell C14' becomes "0.551690284", which falls within the threshold range of the face expansion / contraction ratio.

[0033] Furthermore, as shown in tables T21 and T22 in FIG. 5, the ratios of the syllables "mashita," "," "sowarewa," "screen," "no," "ho," "o," and "" (cells C23 to C30) are "1.5," "0.5," "0.6," "0.6," "0.6," "1," "1.1," and "1.1" (cells R23 to R30), which are not within the threshold range of the body expansion / contraction ratio (greater than 0.75 and less than 1.33).

[0034] Therefore, the setting unit 13 sets a section (cell C23' in table T23) in which the syllables "mashita" and "_" (cells C23 and C24) are merged. The setting unit 13 also sets a section (cell C24' in table T23) in which the syllables "soredemo wa," "kei," "no," "kata," "o," and "_" (cells C25 to C30) are merged. As a result, the ratio in cell C23' becomes "0.973684144," which is within the threshold range of the body expansion / contraction ratio. The ratio in cell C24' becomes "0.760895964," which is within the threshold range of the body expansion / contraction ratio.

[0035] The setting unit 13 outputs the adjusted expansion / contraction sections (tables T13, T23) adjusted for each part.

[0036] Based on the motion data of person A, the conversion unit 14 expands or contracts the motion data to match each section of the voice data of person X, thereby generating motion data of person X. The conversion unit 14 receives as input the motion data of person A, the adjusted expansion / contraction sections T13 and T23 of person A and person B, and the speech times (start and end times for each syllable) of person A and person X. The adjusted expansion / contraction sections T13 and T23 are obtained by adjusting the expansion / contraction rate, syllables, and their sections for each body part by the setting unit 13. The conversion unit 14 outputs the motion data of person X.

[0037] For a first section of the voice data of person X, the conversion unit 14 expands or contracts the motion data of person A in a section corresponding to the first section so that the data fits into the first section, thereby generating motion data for the first section of person X. This is performed for each section of each body part.

[0038] 6 is a diagram illustrating the processing of the conversion unit 14. For example, the required time for the syllable "shouchi" uttered by person A is "0.29" (s) because the start time is "0.104635" and the end time is "0.39498866". In contrast, the required time for the syllable "shouchi" uttered by person X is "0.0033245" and the end time is "0.235607428", so the required time is "0.23" (s).

[0039] The conversion unit 14 compresses the motion data MA1 of person A in the section of the syllable "shouchi" (0.29 seconds) to the time required for person X's syllable "shouchi," 0.23 seconds, and generates the resulting motion data as motion data MX1 of person X. The conversion unit 14 compresses the image intervals so that all images in the motion data MA1 of person A in the section of the syllable "shouchi" (0.29 seconds) fit within 0.23 seconds, and generates the resulting motion data MX1 of person X.

[0040] The conversion unit 14 expands or contracts the motion data of person A for each adjusted expansion or contraction section in the order of utterances "did," "did," "then," etc., in accordance with the speech time of person X, thereby creating motion data of person X.

[0041] For example, when processing the syllable "Soredemo wa" (cell C14') uttered by person A, motion data of person X is generated based on the adjusted section set by the setting unit 13.

[0042] In this case, the required time for the syllable "Soredemo wa" (cell C14') uttered by person A is "1.03" (s), since the start time is "0.9949886" and the end time is "2.0299773". In contrast, the required time for the syllable "Soredemo wa" (cell C14') uttered by person X is "1.090607347" and the end time is "1.661600557", so it is "0.57" (s). The conversion unit 14 shortens the intervals between images so that all images of motion data MA1 of person A in the section ("1.03" (s)) of the syllable "Soredemo wa" (cell C14') fit within "0.57" (s), and generates motion data MX1 for person X.

[0043] The conversion unit 14 generates motion data for person X by expanding or contracting the motion data for person A for each adjusted expansion / contraction section in the order of the utterances "did," "made," "then," etc., in accordance with the speech time of person X. The conversion unit 14 outputs the generated motion data for person X in association with each utterance and in the order of each utterance.

[0044] 7 is a flowchart showing the processing procedure of the conversion process according to the embodiment. The conversion device 10 receives as input pair data of voice data and motion data of person A, voice data of person X for which motion data is to be created, and a threshold range of the expansion / contraction ratio set for each body part.

[0045] In the conversion device 10, the speech recognition unit 11 performs a speech recognition process to generate syllable-based spoken text from the speech data of person A and person X (step S1). The speech recognition unit 11 outputs syllable-based spoken text of person A and person X and the speech times of person A and person X (start time and end time of each syllable).

[0046] The ratio calculation unit 12 performs a ratio calculation process to compare the speech times of person A and person X for each syllable and calculate the ratio (step S2).

[0047] The setting unit 13 performs a setting process to adjust the ratio of each syllable and the syllable for each part of the body for which motion is to be generated so that the ratio of each syllable calculated by the ratio calculation unit 12 falls within the threshold range of the expansion / contraction ratio (step S3).

[0048] The conversion unit 14 performs a conversion process to generate motion data for person X by expanding or contracting the motion data based on the motion data for person A in accordance with each section of the voice data for person X (step S4). The conversion device 10 outputs the motion data for person X generated by the conversion unit 14.

[0049] [Effects of the embodiment] As described above, the conversion device 10 according to the embodiment can generate motions that match the voice of person X, even for person X for whom only voice data exists, from the voice data and motion data of person A. In particular, the conversion device 10 can obtain motion data of person X even if person X is a fictitious character created by voice synthesis or the like.

[0050] Therefore, the conversion device 10 can be utilized as follows: Figures 8 and 9 are diagrams for explaining utilization examples of the embodiment.

[0051] As shown in Figure 8, a conversion device 10 is provided before a motion generation program 20. The conversion device 10 generates motion data of person X, for which only speech data exists, from the speech data and motion data of person A, and inputs the generated motion data to a learning device 21. In the learning phase, the learning device 21 uses the speech data of person X and the motion data generated by the conversion device 10 as training data to execute learning of a learning model 23 of an estimator 22. In the estimation phase, when speech data of person X is input, the estimator 22 can estimate and output motion data of person X that matches this speech.

[0052] In this way, by applying the speech data of person X and the motion data generated by the conversion device 10 to the data before learning as training data creation, it becomes possible to create a learning model that can generate motion data of person X based on the speech data of person X for which no training data exists.

[0053] 9, a conversion device 10 is provided after the motion generation program 30. In the learning phase, a learning device 31 uses the speech data of person A and the motion data of person A as learning data to execute learning of a learning model 33 of an estimator 32. In the estimation phase, when the speech data of person A is input, the estimator 32 estimates and outputs motion data of person A that matches this voice. The conversion device 10 receives input of the motion data of person A estimated by the estimator 32, the speech data of person A, and the speech data of person X, and generates motion data of person X.

[0054] In this way, when the conversion device 10 receives the motion data of person A estimated by the estimator 32 as input, it can also be applied to voice replacement.

[0055] [Example of Motion Data Generation] The processing of the conversion device 10 will be specifically described with reference to Figures 10 to 12. Figures 10 to 12 are diagrams for explaining an example of motion data generation. The graphs in Figures 10 to 12 associate the values ​​of each point of the motion with the content and interval of speech.

[0056] For example, as shown in (1) of Fig. 10, recorded voice data and motion data of person A are prepared. Then, as shown in (2) of Fig. 10, another voice of person X is prepared, which has the same speech content as person A but different speech timing and speech rate. The other voice has the same tone of voice as the voice used in the estimation phase (see Figs. 8 and 9).

[0057] The conversion device 10 adjusts the ratio of each syllable and the syllable section for each part of the body for which motion is generated so that the speech time ratio between person A and person X falls within the threshold range of the expansion / contraction ratio for each syllable.

[0058] Then, as shown in Fig. 11 , the conversion device 10 associates the motion data ((1) in Fig. 11 ) of the recorded data for the same utterance section of the different voice ((2) in Fig. 11 ). For example, the conversion device 10 associates the motion data for the utterance section of the syllable "Today" in the recorded data as the motion data for the utterance section of the syllable "Today" in the different voice. In this case, the conversion device 10 multiplies the motion data for each utterance by [length of the utterance section of the different voice] / [length of the utterance section of the recorded data] in the time axis direction so as to match the utterance section of the different voice, and applies the resulting motion data as the motion data of the different voice.

[0059] The conversion device 10 assumes that each point in the motion data changes linearly, and complements the motion data of a different voice with the recorded data ((1) in Figure 12) as shown in (2) in Figure 12, sampling every 1 / 30 seconds. The conversion device 10 performs this process for the X and Y coordinates of all points.

[0060] [System Configuration of the Embodiment] The components of the conversion device 10 are conceptually functional and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the conversion device 10 is not limited to that shown in the figure, and all or part of the functions can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0061] Furthermore, all or any part of the processes performed by the conversion device 10 may be realized by a CPU and a program analyzed and executed by the CPU. Furthermore, each process performed by the conversion device 10 may be realized as hardware using wired logic.

[0062] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0063] 13 is a diagram showing an example of a computer in which the conversion device 10 is realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0064] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0065] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the conversion device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the conversion device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0066] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0067] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0068] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0069] 10 Conversion device 11 Speech recognition unit 12 Ratio calculation unit 13 Setting unit 14 Conversion unit

Claims

1. A conversion device comprising: a voice recognition unit that generates syllable-based spoken text of the voice data of a first person and the voice data of the second person from the voice data of the first person and the voice data of the second person, respectively; a calculation unit that calculates, for each syllable, a ratio between the speech time of the first person and the speech time of the second person; a setting unit that adjusts the ratio for each syllable and the syllable section for each body part for which motion is generated; and a conversion unit that stretches or shrinks the motion data of the motion data of the first person to match the syllable section of the voice data of the second person, to generate motion data of the second person.

2. The conversion device described in claim 1, characterized in that the setting unit adjusts the ratio for each syllable and the section of the syllable based on a threshold range of the expansion / contraction ratio set for each part of the body for which motion is generated.

3. The conversion device according to claim 2, characterized in that the setting unit expands or contracts the syllable section so that it falls within a threshold range of the expansion / contraction ratio set for each part of the body for which motion is generated.

4. The conversion device described in claim 2, characterized in that the setting unit sets a section in which, for a syllable whose ratio calculated by the calculation unit exceeds the threshold range of the expansion / contraction ratio, subsequent syllables are merged until the ratio calculated by the calculation unit falls within the threshold range of the expansion / contraction ratio.

5. The conversion device according to claim 2, characterized in that the threshold range of the expansion / contraction ratio for the face is wider than that for other parts.

6. The conversion device according to claim 2, characterized in that the threshold range of the expansion / contraction ratio for the body is narrower than that for other parts.

7. A conversion method executed by a conversion device, comprising the steps of: generating syllable-based speech text of the voice data of a first person and the voice data of the second person from the voice data of the first person and the voice data of the second person, respectively; calculating the ratio between the speech time of the first person and the speech time of the second person for each syllable; adjusting the ratio for each syllable and the syllable section for each part of the body for which motion is to be generated; and stretching or shrinking the motion data of the motion data of the first person to match the syllable section of the voice data of the second person, thereby generating motion data of the second person.

8. A conversion program for causing a computer to execute the steps of: generating syllable-based speech text of the voice data of a first person and the voice data of the second person from the voice data of the first person and the voice data of the second person, respectively; calculating for each syllable the ratio between the speech time of the first person and the speech time of the second person; adjusting the ratio for each syllable and the syllable section for each body part for which motion is generated; and stretching or shrinking the motion data of the motion data of the first person to match the syllable section of the voice data of the second person, thereby generating motion data of the second person.

Images