Translation language evaluation device, translation language evaluation system, translation language evaluation method, and program

Abstract

The present invention makes it possible to appropriately evaluate whether or not a translated sentence corresponds to the movements of the mouth of a character. At least one processor (11) generates a level of similarity indicating similarities of the movements of the mouth of the character on the basis of the shape of the mouth of the character corresponding to each of phonemes (33a to 33c) included in a phoneme string (33) before translation and the shape of the mouth of the character corresponding to each of phonemes (34a to 34e, 35a, 35b) included in phoneme strings (34, 35) after the translation.

Description

Translation language evaluation device, translation language evaluation system, translation language evaluation method, and program

[0001] The present disclosure relates to a translation language evaluation device, a translation language evaluation system, a translation language evaluation method, and a program.

[0002] 2. Description of the Related Art In some content, such as games and movies, the audio of the dialogue of characters appearing in the content is sometimes replaced (dubbed) with audio in a translated language (hereinafter also referred to as a translated language).

[0003] When the shape of a character's mouth corresponding to the original sentence before translation is significantly different from the shape of the character's mouth corresponding to the translated sentence, the voice of the translated sentence does not correspond to the movement of the character's mouth, which can cause discomfort to game players and viewers of video works.

[0004] An object of the present disclosure is to enable an appropriate evaluation of whether a translated sentence corresponds to a character's mouth movements.

[0005] A translation language evaluation device according to the present disclosure may include at least one processor. The at least one processor may acquire a pre-translation phoneme sequence indicating a pre-translation phoneme order based on a pre-translation original text, acquire a translated phoneme sequence indicating a post-translation phoneme order based on a translation obtained by translating the language of the pre-translation original text into another language, and generate a similarity indicating a similarity of the character's mouth movements based on a character's mouth shape corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and a character's mouth shape corresponding to each translated phoneme included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether a translation text corresponds to the character's mouth movements.

[0006] The translation language evaluation system according to the present disclosure may include at least one processor. The at least one processor may acquire a pre-translation phoneme sequence indicating the order of phonemes before translation based on an original text before translation, acquire a translated phoneme sequence indicating the order of phonemes after translation based on a translation obtained by translating the language of the original text before translation into another language, and generate a similarity indicating the similarity of the character's mouth movements based on a character's mouth shape corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and a character's mouth shape corresponding to each translated phoneme included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether the translation text corresponds to the character's mouth movements.

[0007] Furthermore, a translation language evaluation method according to the present disclosure may include the steps of: acquiring a pre-translation phoneme sequence indicating the order of phonemes before translation based on an original text before translation; acquiring a translated phoneme sequence indicating the order of phonemes after translation based on a translation obtained by translating the language of the original text before translation into another language; and generating a similarity indicating the similarity of the character's mouth movements based on a character's mouth shape corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and a character's mouth shape corresponding to each of the translated phonemes included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether the translation matches the character's mouth movements.

[0008] Furthermore, a program according to the present disclosure may cause a computer to execute the following steps: acquiring a pre-translation phoneme sequence indicating the order of phonemes before translation, based on an original text before translation; acquiring a translated phoneme sequence indicating the order of phonemes after translation, based on a translation obtained by translating the language of the original text before translation into another language; and generating a similarity indicating the similarity of the character's mouth movements, based on the character's mouth shapes corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to each of the translated phonemes included in the translated phoneme sequence. This makes it possible to use a computer to appropriately evaluate whether a translation matches the character's mouth movements.

[0009] FIG. 1 is a diagram illustrating an example of a hardware configuration of a translation language evaluation method according to an embodiment of the present disclosure. FIG. 2 is a functional block diagram illustrating an example of functions implemented in the translation language evaluation method. FIG. 3 is a diagram illustrating an example of an original sentence, a translation sentence, a phoneme string before translation, and a phoneme string after translation, which are acquired or generated by each function of the translation language evaluation device. FIG. 4 is a diagram illustrating an example of a speech period of each phoneme included in a phoneme string. FIG. 5 is a diagram illustrating an example of a speech period of each phoneme included in a phoneme string. FIG. 6 is a diagram illustrating an example of a method of generating similarity. FIG. 7 is a diagram illustrating a period in which speech periods overlap in the example shown in FIG. 4. FIG. 8 is a diagram illustrating an example of a flow of processing performed in the translation language evaluation device. FIG. 9 is a functional block diagram illustrating another example of functions implemented in the translation language evaluation device. FIG. 10 is a diagram illustrating an example of a speech period of each phoneme included in a phoneme string.

[0010] [1. First Embodiment] An example of an embodiment of the present disclosure will now be described with reference to the drawings. A translation language evaluation device according to this embodiment is designed to appropriately evaluate whether a translation corresponds to the movement of the character's mouth when speaking an original sentence (whether the voice when speaking the translation matches the movement of the character's mouth when speaking the original sentence), based on the shape of the character's mouth when speaking an original sentence before translation and the shape of the character's mouth when speaking a translated sentence obtained by translating the original sentence.

[0011] [1-1. Hardware Configuration] Fig. 1 is a diagram showing an example of the hardware configuration of a translation language evaluation device 10 (translation language evaluation system). The translation language evaluation device 10 is a computer such as a personal computer, and as shown in Fig. 1, may include a processor 11, a storage unit 12, a communication unit 13, a display unit 14, and an operation unit 15.

[0012] The processor 11 is a program-controlled device such as a CPU (Central Processing Unit) that operates according to a program installed in the translation language evaluation device 10. The storage unit 12 is a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), or an HDD (Hard Disk Drive). The storage unit 12 stores data such as programs executed by the processor 11. The communication unit 13 is a communication interface such as a network board. The display unit 14 is a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, and displays various images according to instructions from the processor 11. The operation unit 15 is a user interface such as a keyboard, mouse, or game controller that accepts user operation input and outputs signals indicating the content of the input to the processor 11.

[0013] The translation language evaluation device 10 may also include an optical disc drive for reading optical discs, a video output terminal such as DisplayPort (registered trademark), a data input / output terminal such as USB (Universal Serial Bus), and an audio output terminal such as a speaker or earphone jack.

[0014] [1-2. Functional Blocks] FIG. 2 is a functional block diagram showing an example of functions implemented in the translation language evaluation device 10 (translation language evaluation system). As shown in FIG. 2, the translation language evaluation device 10 may functionally include an original sentence acquisition unit 21, a translation sentence acquisition unit 22, a pre-translation phoneme sequence acquisition unit 23, a post-translation phoneme sequence acquisition unit 24, a pre-translation utterance period determination unit 25, a post-translation utterance period determination unit 26, and a similarity generation unit 27. These functions may be realized primarily by the processor 11, or may be realized by multiple processors including the processor 11. Furthermore, the translation language evaluation device 10 does not need to implement all of the functions shown in FIG. 2, and functions other than the functions shown in FIG. 2 may also be implemented.

[0015] [1-2-1. Original Text Acquisition Unit, Translation Text Acquisition Unit] The original text acquisition unit 21 acquires an original text before translation. The translation text acquisition unit 22 acquires a translation obtained by translating the original text (e.g., an original text in English) acquired by the original text acquisition unit 21 into another language (e.g., Japanese). The original text acquisition unit 21 may acquire text data as the original text. Similarly, the translation text acquisition unit 22 may acquire text data as the translation text. The translation text acquisition unit 22 may acquire multiple translation candidates as the translation text.

[0016] [1-2-2. Pre-translation Phoneme Sequence Acquisition Unit, Post-translation Phoneme Sequence Acquisition Unit] The pre-translation phoneme sequence acquisition unit 23 acquires a pre-translation phoneme sequence indicating the order of phonemes before translation, based on the original text before translation acquired by the original text acquisition unit 21. The post-translation phoneme sequence acquisition unit 24 acquires a translated phoneme sequence indicating the order of phonemes after translation, based on the translation acquired by the translation text acquisition unit 22.

[0017] Each phoneme included in the phoneme sequence acquired by the pre-translation phoneme sequence acquisition unit 23 and the post-translation phoneme sequence acquisition unit 24 may be information indicated by a symbol such as the International Phonetic Alphabet. Also, each phoneme included in the phoneme sequence may be information indicating the shape of the mouth of the character corresponding to that phoneme (for example, an image of the mouth or a feature of the mouth shape).

[0018] 3 is a diagram showing an example of an original text, a translation text, a phoneme string before translation, and a phoneme string after translation, which are acquired or generated by each function of the translation language evaluation device 10. In the example shown in FIG. 3, the original text acquisition unit 21 acquires an original text 31 in English, "Hello." In addition, in the example shown in FIG. 3, the translation text acquisition unit 22 acquires a translation text 32a of "Hello (Konnichiwa)" and a translation text 32b of "Yaa (Yaa)" as translation texts obtained by translating the original English text 31 into Japanese.

[0019] 3, the pre-translation phoneme sequence acquisition unit 23 acquires a pre-translation phoneme sequence 33 indicating the order of phonemes in the original sentence 31. In the pre-translation phoneme sequence 33, three pre-translation phonemes 33a to 33c are arranged in this order. In the example shown in FIG. 3, the post-translation phoneme sequence acquisition unit 24 acquires a translated phoneme sequence 34 indicating the order of phonemes in the translated sentence 32a, and a translated phoneme sequence 35 indicating the order of phonemes in the translated sentence 32b. In the translated phoneme sequence 34, five translated phonemes 34a to 34e are arranged in this order. In the translated phoneme sequence 35, two translated phonemes 35a and 35b are arranged in this order.

[0020] In the following description, the pre-translation phoneme sequence 33 and the translated phoneme sequences 34 and 35 may be simply referred to as phoneme sequences. In FIG. 3, the phonemes 33a to 33c, 34a to 34e, 35a, and 35b included in the phoneme sequences 33 to 35 are shown with mouth shapes, but information about each phoneme may be represented by the International Phonetic Alphabet or the like. Also, in FIG. 3, the phonemes 33a to 33c, 34a to 34e, 35a, and 35b may be different phonemes. Of these phonemes, the phonemes 33a, 34e, 35a, and 35b may be similar phonemes that do not match. Similarly, the phonemes 34c and 34d may be similar phonemes.

[0021] The pre-translation phoneme sequence acquisition unit 23 may acquire the phoneme sequence 33 by generating phonemes (phonemes 33a to 33c) based on the original sentence 31. Alternatively, the pre-translation phoneme sequence acquisition unit 23 may acquire a phoneme sequence stored in the storage unit 12 or an external storage device in association with the original sentence 31 as the phoneme sequence 33 of the original sentence 31. Alternatively, the pre-translation phoneme sequence acquisition unit 23 may acquire the phoneme sequence 33 by receiving phoneme sequence information via the communication unit 13. Similarly, the post-translation phoneme sequence acquisition unit 24 may acquire the phoneme sequences 34 and 35 by generating phonemes based on the translation sentences 32a and 32b, or may acquire a phoneme sequence stored in association with the translation sentences 32a and 32b as the phoneme sequences 34 and 35.

[0022] [1-2-3. Pre-translation Speech Period Determination Unit, Post-translation Speech Period Determination Unit] The pre-translation utterance period determination unit 25 determines the utterance period of each pre-translation phoneme included in the pre-translation phoneme sequence acquired by the pre-translation phoneme sequence acquisition unit 23. The post-translation utterance period determination unit 26 determines the utterance period of each post-translation phoneme included in the post-translation phoneme sequence acquired by the post-translation phoneme sequence acquisition unit 24.

[0023] Fig. 4 is a diagram showing an example of the utterance periods of each phoneme included in a phoneme sequence. In the example shown in Fig. 4, the pre-translation utterance period determination unit 25 determines utterance periods T1a to T1c corresponding to phonemes 33a to 33c included in a pre-translation phoneme sequence 33. In addition, in the example shown in Fig. 4, the post-translation utterance period determination unit 26 determines utterance periods T2a to T2e corresponding to phonemes 34a to 34e included in a translated phoneme sequence 34, and utterance periods T3a and T3b corresponding to phonemes 35a and 35b included in a translated phoneme sequence 35.

[0024] The pre-translation utterance period determination unit 25 may determine periods of the same length as the utterance periods of each pre-translation phoneme included in the pre-translation phoneme sequence. Similarly, the post-translation utterance period determination unit 26 may determine periods of the same length as the utterance periods of each translated phoneme included in the translated phoneme sequence.

[0025] 4, the pre-translation utterance period determination unit 25 determines utterance periods T1a to T1c corresponding to the phonemes 33a to 33c by dividing the predetermined period T by "3," which is the number of phonemes 33a to 33c included in the pre-translation phoneme sequence 33. Here, the utterance periods T1a to T1c are of the same length. Also, in the example shown in FIG. 4, the post-translation utterance period determination unit 26 determines utterance periods T2a to T2c of the same length corresponding to the phonemes 34a to 34e by dividing the predetermined period T by "5," which is the number of phonemes 34a to 34e included in the translated phoneme sequence 34, and determines utterance periods T3a and T3b of the same length corresponding to the phonemes 35a and 35b by dividing the predetermined period T by "2," which is the number of phonemes 35a and 35b included in the translated phoneme sequence 35.

[0026] 5A and 5B are diagrams showing examples of the utterance periods of each phoneme included in a phoneme sequence. The pre-translation utterance period determination unit 25 may determine, as the utterance period of each pre-translation phoneme included in the pre-translation phoneme sequence, a period having a length corresponding to the number of translated phonemes included in the translated phoneme sequence. Similarly, the post-translation utterance period determination unit 26 may determine, as the utterance period of each translated phoneme included in the translated phoneme sequence, a period having a length corresponding to the number of pre-translation phonemes included in the pre-translation phoneme sequence.

[0027] 5A, the pre-translation utterance period determination unit 25 determines utterance periods T4a to T4c of the same length corresponding to the phonemes 33a to 33c by multiplying the predetermined period ΔT by "5," which is the number of phonemes 34a to 34e included in the translated phoneme sequence 34. Also, in the example shown in FIG. 5A, the post-translation utterance period determination unit 26 determines utterance periods T5a to T5e of the same length corresponding to the phonemes 34a to 34e by multiplying the predetermined period ΔT by "3," which is the number of phonemes 33a to 33c included in the pre-translation phoneme sequence 33. Here, the period T4 obtained by continuously connecting the utterance periods T4a to T4c may be the same as the period T5 obtained by continuously connecting the utterance periods T5a to T5e.

[0028] 5B, the pre-translation utterance period determination unit 25 determines utterance periods T6a to T6c of the same length corresponding to the phonemes 33a to 33c by multiplying the predetermined period ΔT by "2," which is the number of phonemes 35a and 35b included in the translated phoneme sequence 35. Also, in the example shown in FIG. 5B, the post-translation utterance period determination unit 26 determines utterance periods T7a and T7b of the same length corresponding to the phonemes 35a and 35b by multiplying the predetermined period ΔT by "3," which is the number of phonemes 33a to 33c included in the pre-translation phoneme sequence 33. The period T6 obtained by continuously connecting the utterance periods T6a to T6c may be the same as the period T7 obtained by continuously connecting the utterance periods T7a and T7b.

[0029] [1-2-4. Similarity Generation Unit] The similarity generation unit 27 generates a similarity indicating the similarity of the character's mouth movements based on the character's mouth shapes corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to each translated phoneme included in the translated phoneme sequence. In this way, it is possible to appropriately evaluate whether the translated sentence corresponds to the character's mouth movements.

[0030] 6 is a diagram illustrating an example of a similarity generation method. In the example illustrated in FIG. 6, the similarity generation unit 27 calculates the similarity of the character's mouth shape by comparing the positions of mouth feature points 41a to 41h corresponding to phoneme 33a included in the pre-translation phoneme sequence 33 with the positions of mouth feature points 42a to 42h corresponding to phoneme 34a included in the translated phoneme sequence 34. The similarity generation unit 27 may calculate the similarity of the mouth shape by, for example, calculating the distance between feature point 41a and feature point 42a. Based on the similarity of the mouth shapes calculated in this manner, the similarity generation unit 27 may generate a similarity that indicates the similarity of the character's mouth movements indicated by the phoneme sequences before and after translation.

[0031] Furthermore, the similarity generation unit 27 may obtain the similarity of the mouth shapes of the characters stored in the storage unit 12 or an external storage device in association with the phonemes before and after translation based on the phonemes before and after translation. Furthermore, the similarity generation unit 27 may receive the similarity of the mouth shapes via the communication unit 13.

[0032] The similarity generating unit 27 may also generate similarities based on the mouth shapes of characters indicated by pre-translation phonemes and translated phonemes whose vocalization periods overlap.

[0033] 7 is a diagram showing periods in which the utterance periods overlap in the example shown in FIG. 4. In the examples shown in FIGS. 4 and 7, the similarity generation unit 27 may calculate the similarity of the mouth shapes shown by the phonemes 33a and 34a in a period T11 in which the utterance period T1a of the pre-translation phoneme 33a and the utterance period T2a of the translated phoneme 34a overlap. The similarity generation unit 27 may also calculate the similarity of the mouth shapes shown by the phonemes 33a and 34b in a period T12 in which the utterance periods of the phonemes 33a and 34b overlap, the similarity of the mouth shapes shown by the phonemes 33b and 34b in a period T13 in which the utterance periods of the phonemes 33b and 34b overlap, and the similarity of the mouth shapes shown by the phonemes 33b and 34c in a period T14 in which the utterance periods of the phonemes 33b and 34c overlap. The similarity generation unit 27 may calculate the similarity of the mouth shapes indicated by the phonemes 33b and 34d in a period T15 when the utterance periods of the phonemes 33b and 34d overlap, the similarity of the mouth shapes indicated by the phonemes 33c and 34d in a period T16 when the utterance periods of the phonemes 33c and 34d overlap, and the similarity of the mouth shapes indicated by the phonemes 33c and 34d in a period T17 when the utterance periods of the phonemes 33c and 34e overlap. The similarity generation unit 27 may generate a similarity indicating the similarity of the character's mouth movements by multiplying the multiple (seven in the example shown in FIGS. 4 and 7 ) numerical values ​​indicating the degree of similarity calculated as above by each of the periods T11 to T17. This allows for an appropriate evaluation of whether the translated sentence corresponds to the character's mouth movements.

[0034] 5A and 5B, the similarity generation unit 27 may calculate, for each predetermined period (e.g., period ΔT in FIGS. 5A and 5B or a period shorter than period ΔT), the similarity between the mouth shape indicated by a pre-translation phoneme (e.g., any of phonemes 33a to 33c shown in FIGS. 5A and 5B) whose vocalization period overlaps with the period, and the mouth shape indicated by a translated phoneme (e.g., any of phonemes 34a to 34e shown in FIG. 5A or any of phonemes 35a and 35b shown in FIG. 5B) whose vocalization period overlaps with the period. In this case, the similarity generation unit 27 may generate a similarity indicating the similarity of the character's mouth movements based on the cumulative total of multiple similarity-indicating values ​​calculated for each predetermined period. This also makes it possible to appropriately evaluate whether a translation corresponds to the character's mouth movements.

[0035] 8 is a diagram showing an example of the flow of processing performed by the translation language evaluation device 10 (translation language evaluation system). The flow of processing performed by the translation language evaluation device 10 will be described below with reference to FIG.

[0036] As shown in FIG. 8, the original text acquisition unit 21 and the translation text acquisition unit 22 acquire an original text before translation (e.g., the English original text 31 shown in FIG. 3) and a translation of the original text into another language (e.g., the Japanese translations 32a and 32b shown in FIG. 3) (step S101).

[0037] Next, the pre-translation phoneme sequence acquisition unit 23 acquires a pre-translation phoneme sequence indicating the order of phonemes in the original sentence based on the original sentence acquired in step S101 (step S102). Also, the post-translation phoneme sequence acquisition unit 24 acquires a post-translation phoneme sequence indicating the order of phonemes in the translation based on the translation acquired in step S101 (step S103). Note that the order of executing steps S102 and S103 may be reversed.

[0038] In step S102, the pre-translation phoneme sequence acquisition unit 23 may generate phonemes (e.g., phonemes 33a to 33c shown in FIG. 3) as a pre-translation phoneme sequence based on the original sentence acquired in step S101, or may acquire a phoneme sequence stored in association with the original sentence in the storage unit 12 or an external storage device. Similarly, in step S103, the post-translation phoneme sequence acquisition unit 24 may generate phonemes (e.g., phonemes 34a to 34e, 35a, and 35b shown in FIG. 3) as a post-translation phoneme sequence based on the translation sentence acquired in step S101, or may acquire a phoneme sequence stored in association with the translation sentence in the storage unit 12 or an external storage device.

[0039] Next, the pre-translation utterance period determination unit 25 determines the utterance period of each phoneme included in the pre-translation phoneme sequence obtained in step S102 (step S104). The post-translation utterance period determination unit 26 determines the utterance period of each phoneme included in the translated phoneme sequence obtained in step S103 (step S105). Note that the order of executing steps S104 and S105 may be reversed.

[0040] In step S104, the pre-translation utterance period determination unit 25 may determine periods of the same length as the utterance periods of each phoneme included in the pre-translation phoneme sequence acquired in step S102. Similarly, in step S105, the post-translation utterance period determination unit 26 may determine periods of the same length as the utterance periods of each phoneme included in the translated phoneme sequence acquired in step S103.

[0041] In step S104, the pre-translation utterance period determination unit 25 may determine the utterance period corresponding to each pre-translation phoneme by dividing a predetermined period T by the number of phonemes included in the pre-translation phoneme sequence acquired in step S102 (e.g., phonemes 33a to 33c included in the phoneme sequence 33 shown in FIG. 4), as shown in FIG. 4. Similarly, in step S105, the post-translation utterance period determination unit 26 may determine the utterance period corresponding to each post-translation phoneme by dividing a predetermined period T by the number of phonemes included in the post-translation phoneme sequence acquired in step S103 (e.g., phonemes 34a to 34e included in the phoneme sequence 34 shown in FIG. 4).

[0042] 5A and 5B, in step S104, the pre-translation speaking period determination unit 25 may determine the speaking period corresponding to each phoneme included in the pre-translation phoneme sequence acquired in step S102 by multiplying a predetermined period ΔT by the number of phonemes included in the post-translation phoneme sequence acquired in step S103 (e.g., 34a to 34e included in phoneme sequence 34 shown in FIG. 5A). Similarly, in step S105, the post-translation speaking period determination unit 26 may determine the speaking period corresponding to each phoneme included in the post-translation phoneme sequence by multiplying a predetermined period ΔT by the number of phonemes included in the pre-translation phoneme sequence (e.g., 33a to 33c included in phoneme sequence 33 shown in FIG. 5A).

[0043] Next, the similarity generation unit 27 generates a similarity indicating the similarity of the character's mouth movements based on the character's mouth shapes corresponding to each phoneme before and after translation (step S106), and the translation language evaluation device 10 ends the processing.

[0044] 6, the similarity generation unit 27 may calculate the similarity of the character's mouth shape by comparing the positions of mouth feature points 41a to 41h corresponding to phoneme 33a included in the pre-translation phoneme sequence 33 acquired in step S102 with the positions of mouth feature points 42a to 42h corresponding to phoneme 34a included in the translated phoneme sequence 34 acquired in step S103. Based on the similarity of the mouth shapes calculated in this way, the similarity generation unit 27 may generate a similarity that indicates the similarity of the character's mouth movements indicated by the phoneme sequences before and after translation.

[0045] Furthermore, in step S106, the similarity generation unit 27 may obtain the similarity of the mouth shapes of the characters stored in the memory unit 12 or an external storage device in association with the phonemes contained in the pre-translation phoneme sequence obtained in step S102 and the phonemes contained in the translated phoneme sequence obtained in step S103.

[0046] Furthermore, in step S106, the similarity generation unit 27 may generate a similarity indicating the similarity in the character's mouth movements based on the mouth shapes of the character indicated by each of the pre-translation phonemes and the translated phonemes whose vocalization periods overlap. In step S106, the similarity generation unit 27 may generate a similarity indicating the similarity in the mouth movements based on the similarity in the mouth shapes calculated for each of the periods T11 to T17 in which the pre-translation and post-translation phonemes overlap as shown in FIG. 7. In this case, the similarity indicating the similarity in the character's mouth movements may be generated by multiplying each of the periods T11 to T17 by a numerical value indicating the similarity in the mouth shapes indicated by the pre-translation and post-translation phonemes in that period.

[0047] Furthermore, in step S106, the similarity generation unit 27 may, for example, calculate a numerical value indicating the similarity between the mouth shapes indicated by each phoneme before and after translation whose vocalization period overlaps with the predetermined period (for example, period ΔT in Figures 5A and 5B or a period shorter than period ΔT), and generate a similarity indicating the similarity of the character's mouth movements based on the cumulative total of the numerical values ​​calculated in this way.

[0048] As described above, the similarity generation unit 27 generates a similarity indicating the similarity of the character's mouth movements based on the character's mouth shapes corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to each translated phoneme included in the translated phoneme sequence. In this way, it becomes possible to appropriately evaluate whether the translation corresponds to the character's mouth movements.

[0049] In this embodiment, the pre-translation utterance period determination unit 25 determines periods of the same length as the utterance periods of each phoneme included in the pre-translation phoneme sequence. Similarly, the post-translation utterance period determination unit 26 determines periods of the same length as the utterance periods of each phoneme included in the translated phoneme sequence. The similarity generation unit 27 then generates a similarity based on the mouth shapes of the character indicated by the pre-translation phonemes and the translated phonemes whose utterance periods overlap. This makes it possible to appropriately evaluate whether the translated sentence corresponds to the mouth movements of the character.

[0050] The present invention is not limited to the above-described embodiments. For example, variations of the above-described embodiments may also be included within the technical scope of the present invention.

[0051] 2. Second Embodiment Fig. 9 is a functional block diagram showing another example of functions implemented in the translation language evaluation device 10 (translation language evaluation system). In addition to the functions described in the first embodiment, the translation language evaluation device 10 may have an original text speech acquisition unit 28 and a translation text speech acquisition unit 29 shown in Fig. 9. The original text speech acquisition unit 28 and the translation text speech acquisition unit 29 may be realized mainly by the processor 11.

[0052] The original text speech acquisition unit 28 acquires the speech of the original text acquired by the original text acquisition unit 21. The original text speech acquisition unit 28 acquires, for example, the speech of the English original text 31 shown in Fig. 3. The translation text speech acquisition unit 29 acquires the speech of the translation text acquired by the translation text acquisition unit 22. The translation text speech acquisition unit 29 acquires, for example, the speech of the Japanese translation texts 32a and 32b shown in Fig. 3.

[0053] FIG. 10 is a diagram showing an example of the utterance periods of each phoneme included in a phoneme sequence. As shown in FIG. 10, the pre-translation utterance period determiner 25 may determine utterance periods T8a to T8c of each phoneme 33a to 33c included in the pre-translation phoneme sequence 33 acquired by the pre-translation phoneme sequence acquirer 23 based on the speech of the original sentence 31 (see FIG. 3) acquired by the original sentence speech acquirer 28. The pre-translation utterance period determiner 25 may also determine utterance periods T9a to T9e of each phoneme 34a to 34e included in the translated phoneme sequence 34 acquired by the translated phoneme sequence acquirer 24 based on the speech of the translated sentence 32a (see FIG. 3) acquired by the translated sentence speech acquirer 29. Similarly, the pre-translation utterance period determiner 25 may determine utterance periods T10a and T10b of each phoneme 35a and 35b included in the translated phoneme sequence 35 based on the speech of the translated sentence 32b (see FIG. 3).

[0054] The pre-translation utterance period determination unit 25 and the post-translation utterance period determination unit 26 may determine the utterance period of each phoneme included in the phoneme sequence by analyzing the speech. For example, the post-translation utterance period determination unit 26 may edit the speech of the translated sentence 32a so that the utterance period T9 of the speech of the translated sentence 32a coincides with the utterance period T8 of the speech of the original sentence 31, and then determine the utterance periods T9a to T9e of each phoneme 34a to 34e included in the translated phoneme sequence 34 based on the edited speech. Alternatively, the pre-translation utterance period determination unit 25 may edit the speech of the original sentence 31 so that the utterance period T9 of the speech of the original sentence 31 coincides with the utterance period T8 of the speech of the translated sentence 32a, and then determine the utterance periods T8a to T8c of each phoneme 33a to 33c included in the pre-translation phoneme sequence 33 based on the edited speech.

[0055] In this embodiment, the similarity generation unit 27 may also generate a similarity indicating the similarity of the character's mouth movements based on the mouth shapes of the character indicated by phonemes before and after translation whose vocalization periods overlap. The similarity generation unit 27 may generate a similarity indicating the similarity of the character's mouth movements by multiplying a numerical value indicating the similarity of the mouth shapes of phonemes before and after translation during a vocalization period by the period. Alternatively, the similarity generation unit 27 may calculate a numerical value indicating the similarity of the mouth shapes of each phoneme before and after translation whose vocalization period overlaps with the period (e.g., period ΔT in FIGS. 5A and 5B or a period shorter than period ΔT), for each predetermined period, and generate a similarity indicating the similarity of the character's mouth movements based on the cumulative total of the multiple numerical values ​​thus calculated. This makes it possible to more appropriately evaluate whether a translation corresponds to the character's mouth movements.

[0056] Furthermore, for example, if the speech period T10 of the speech of the translated sentence 32b is shorter than the speech period T8 of the speech of the original sentence 31, the post-translation speech period determination unit 26 may determine the period from the end of speech period T10 to the end of speech period T8 as the period of a predetermined phoneme after translation (e.g., a phoneme indicating that the character's mouth is closed). In this case, the similarity generation unit 27 may generate a similarity indicating the similarity of the character's mouth movements based on the shape of the character's mouth indicated by the pre-translation phoneme (e.g., phoneme 33c in FIG. 10 ) during the period from the end of speech period T10 to the end of speech period T8 and the predetermined phoneme after translation during that period. This also makes it possible to more appropriately evaluate whether the translation corresponds to the character's mouth movements.

[0057] [3. Summary] (1) As described above, the translation language evaluation device 10 described in the present disclosure may include at least one processor (e.g., processor 11). The at least one processor may acquire a pre-translation phoneme sequence (e.g., phoneme sequence 33 shown in FIG. 3 ) indicating the order of phonemes before translation based on an original text before translation (e.g., original text 31 shown in FIG. 3 ). The at least one processor may acquire a translated phoneme sequence (e.g., phoneme sequences 34 and 35 shown in FIG. 3 ) indicating the order of phonemes after translation based on a translation obtained by translating the language of the original text before translation into another language (e.g., translated texts 32a and 32b shown in FIG. 3 ). The at least one processor may generate a similarity indicating the similarity of the character's mouth movements based on a mouth shape of the character corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and a mouth shape of the character corresponding to each translated phoneme included in the translated phoneme sequence. By doing so, it becomes possible to properly evaluate whether the translated text corresponds to the mouth movements of the character.

[0058] (6) The translation language evaluation system described in the present disclosure may also include at least one processor (e.g., processor 11). The at least one processor may acquire a pre-translation phoneme sequence (e.g., phoneme sequence 33 shown in FIG. 3 ) indicating the order of phonemes before translation based on an original text before translation (e.g., original text 31 shown in FIG. 3 ). The at least one processor may acquire a translated phoneme sequence (e.g., phoneme sequences 34 and 35 shown in FIG. 3 ) indicating the order of phonemes after translation based on a translation obtained by translating the language of the original text before translation into another language (e.g., translated texts 32a and 32b shown in FIG. 3 ). The at least one processor may generate a similarity score indicating the similarity of the character's mouth movements based on the character's mouth shapes corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to each translated phoneme included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether a translation matches the character's mouth movements.

[0059] (7) The translation language evaluation method described in the present disclosure may include the steps of: acquiring a pre-translation phoneme sequence (e.g., phoneme sequence 33 shown in FIG. 3 ) indicating the order of phonemes before translation based on an original text before translation (e.g., original text 31 shown in FIG. 3 ); acquiring a translated phoneme sequence (e.g., phoneme sequences 34 and 35 shown in FIG. 3 ) indicating the order of phonemes after translation based on a translation obtained by translating the language of the original text before translation into another language (e.g., translated texts 32 a and 32 b shown in FIG. 3 ); and generating a similarity score indicating the similarity of the character's mouth movements based on the character's mouth shapes corresponding to the pre-translation phonemes included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to the translated phonemes included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether the translation corresponds to the character's mouth movements.

[0060] (8) The program described in the present disclosure also causes the translation language evaluation device 10, which is a computer, to execute the following steps: acquire a pre-translation phoneme sequence (e.g., phoneme sequence 33 shown in FIG. 3 ) indicating the order of phonemes before translation, based on an original text before translation (e.g., original text 31 shown in FIG. 3 ); acquire a translated phoneme sequence (e.g., phoneme sequences 34 and 35 shown in FIG. 3 ) indicating the order of phonemes after translation, based on a translation obtained by translating the language of the original text before translation into another language (e.g., translated texts 32 a and 32 b shown in FIG. 3 ); and generate a similarity score indicating the similarity of the character's mouth movements, based on the character's mouth shapes corresponding to each pre-translation phoneme included in the pre-translation phoneme sequence and the character's mouth shapes corresponding to each translated phoneme included in the translated phoneme sequence. This makes it possible to appropriately evaluate whether a translation corresponds to a character's mouth movements.

[0061] (2) In the translation language evaluation device 10 described in (1) above, the at least one processor may determine the utterance period of each of the pre-translation phonemes included in the pre-translation phoneme sequence (e.g., periods T1a to T1c shown in FIG. 4, periods T4a to T4c shown in FIG. 5A, periods T6a to T6c shown in FIG. 5B, and periods T8a to T8c shown in FIG. 10). The at least one processor may determine the utterance period of each of the translated phonemes included in the translated phoneme sequence (e.g., periods T2a to T2e, T3a, and T3b shown in FIG. 4, periods T4a to T4e shown in FIG. 5A, periods T7a and T7b shown in FIG. 5B, and periods T9a to T9e, T10a, and T10b shown in FIG. 10). The at least one processor may then generate the similarity based on the shape of the character's mouth indicated by each of the pre-translation phonemes and the translated phonemes whose utterance periods overlap.

[0062] (3) In the translation language evaluation device 10 described in (2) above, the at least one processor may determine periods of the same length (e.g., periods T1a to T1c shown in FIG. 4, periods T4a to T4c shown in FIG. 5A, and periods T6a to T6c shown in FIG. 5B) as the utterance periods of each of the pre-translation phonemes included in the pre-translation phoneme sequence. The at least one processor may determine periods of the same length (e.g., periods T2a to T2e, T3a, and T3b shown in FIG. 4, periods T4a to T4e shown in FIG. 5A, and periods T7a and T7b shown in FIG. 5B) as the utterance periods of each of the translated phonemes included in the translated phoneme sequence.

[0063] (4) In the translation language evaluation device 10 described in (3) above, the at least one processor may determine, as the pronunciation period of each pre-translation phoneme included in the pre-translation phoneme sequence, a period having a length corresponding to the number of post-translation phonemes included in the post-translation phoneme sequence (for example, periods T4a to T4c shown in FIG. 5A or periods T6a to T6c shown in FIG. 5B). The at least one processor may determine, as the pronunciation period of each post-translation phoneme included in the post-translation phoneme sequence, a period having a length corresponding to the number of pre-translation phonemes included in the pre-translation phoneme sequence (for example, periods T4a to T4e shown in FIG. 5A or periods T7a and T7b shown in FIG. 5B).

[0064] (5) In the translation language evaluation device 10 described in (2) above, the at least one processor may acquire speech of the original sentence. The at least one processor may acquire speech of the translated sentence. The at least one processor may determine, based on the speech of the original sentence, a speech period of each of the pre-translation phonemes included in the pre-translation phoneme sequence (e.g., periods T8a to T8c shown in FIG. 10 ). The at least one processor may determine, based on the speech of the translated sentence, a speech period of each of the translated phonemes included in the translated phoneme sequence (periods T9a to T9e, T10a, and T10b shown in FIG. 10 ).

Claims

1. A translation language evaluation device including at least one processor, wherein the at least one processor: obtains a pre-translation phoneme sequence indicating an order of phonemes before translation, based on an original text before translation; obtains a translated phoneme sequence indicating an order of phonemes after translation, based on a translation text obtained by translating the language of the original text before translation into another language; and generates a similarity indicating a similarity in the mouth movements of a character, based on the mouth shapes of the character corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and the mouth shapes of the character corresponding to each of the translated phonemes included in the translated phoneme sequence.

2. A translation language evaluation device as described in claim 1, wherein the at least one processor determines a speech period of each of the pre-translation phonemes included in the pre-translation phoneme string, determines a speech period of each of the translated phonemes included in the translated phoneme string, and generates the similarity based on the shape of the character's mouth indicated by each of the pre-translation phonemes and the translated phonemes whose speech periods overlap.

3. A translation language evaluation device according to claim 2, wherein the at least one processor determines periods of equal length as the speech periods of each of the pre-translation phonemes included in the pre-translation phoneme string, and determines periods of equal length as the speech periods of each of the translated phonemes included in the translated phoneme string.

4. A translated language evaluation device as described in claim 3, wherein the at least one processor determines, as the pronunciation period of each pre-translation phoneme included in the pre-translation phoneme string, a period having a length corresponding to the number of translated phonemes included in the translated phoneme string, and determines, as the pronunciation period of each translated phoneme included in the translated phoneme string, a period having a length corresponding to the number of pre-translation phonemes included in the pre-translation phoneme string.

5. A translation language evaluation device as described in claim 2, wherein the at least one processor acquires a speech of the original sentence, acquires a speech of the translated sentence, determines a speech period of each of the pre-translation phonemes included in the pre-translation phoneme sequence based on the speech of the original sentence, and determines a speech period of each of the translated phonemes included in the translated phoneme sequence based on the speech of the translated sentence.

6. A translation language evaluation system including at least one processor, wherein the at least one processor: obtains a pre-translation phoneme sequence indicating an order of phonemes before translation based on an original text before translation; obtains a translated phoneme sequence indicating an order of phonemes after translation based on a translation text obtained by translating the language of the original text before translation into another language; and generates a similarity indicating a similarity in the mouth movements of a character based on the mouth shapes of the character corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and the mouth shapes of the character corresponding to each of the translated phonemes included in the translated phoneme sequence.

7. A translation language evaluation method comprising the steps of: obtaining a pre-translation phoneme sequence indicating an order of phonemes before translation based on an original text before translation; obtaining a translated phoneme sequence indicating an order of phonemes after translation based on a translation text obtained by translating the language of the original text before translation into another language; and generating a similarity indicating a similarity in the mouth movements of a character based on the mouth shapes of the character corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and the mouth shapes of the character corresponding to each of the translated phonemes included in the translated phoneme sequence.

8. A program that causes a computer to execute the steps of: obtaining a pre-translation phoneme sequence indicating the order of phonemes before translation, based on an original text before translation; obtaining a translated phoneme sequence indicating the order of phonemes after translation, based on a translation obtained by translating the language of the original text before translation into another language; and generating a similarity indicating the similarity of the mouth movements of a character, based on the mouth shapes of the character corresponding to each of the pre-translation phonemes included in the pre-translation phoneme sequence and the mouth shapes of the character corresponding to each of the translated phonemes included in the translated phoneme sequence.

Images