Moving image generation system, moving image generation method, and program

Abstract

Provided are a moving image generation system, a moving image generation method, and a program in which a translated subtitle is substituted at an appropriate timing according to a translated voice. A moving image acquisition unit (12) acquires a moving image including a first voice indicating a first uttered sentence in a first language. A voice output timing acquisition unit (14) acquires an output timing in the moving image with the first voice. A subtitle generation unit (40) generates a leading portion subtitle and a subsequent portion subtitle of a second uttered sentence on the basis of the second uttered sentence corresponding to the first uttered sentence. A required time acquisition unit (42b) acquires the time required for voice output of the leading portion subtitle. A voice acquisition unit (30) acquires a second voice corresponding to the first voice. A moving image generation unit (50) synthesizes the second voice, the leading portion subtitle, and the subsequent portion subtitle with the moving image such that the second voice and the leading portion subtitle are outputted at the output timing of the first voice and the subsequent portion subtitle is outputted at a timing when the required time has elapsed since the output timing of the first voice.

Description

Video generation system, video generation method, and program

[0001] The present invention relates to a video generation system, a video generation method, and a program.

[0002] There is a technology for generating subtitles through speech recognition. Patent Document 1 discloses a web conferencing system in which a sender transcribes speech, sends video (images and speech) and the transcribed text data to a receiver, and the receiver can generate subtitles in a language of their choice.

[0003] Japanese Patent Application Laid-Open No. 2023-025837

[0004] The inventors of the present application are considering a system that translates the words spoken in a video into another language, displays the resulting text as subtitles (translated subtitles) within the video, and outputs the audio (translated audio) within the video. Such a system is expected to make it easier for viewers to understand the content of videos in a foreign language. However, if the translated audio and translated subtitles are not output in sync, viewers may become confused.

[0005] The present invention has been made in consideration of the above-mentioned problems, and one of its objectives is to provide a video generation system, a video generation method, and a program that switch translated subtitles at appropriate timing in accordance with translated audio.

[0006] (1) A moving image generation system according to the present invention includes: moving image acquisition means for acquiring moving image including first audio indicating a first spoken sentence in a first language; audio output timing acquisition means for acquiring an output timing of the first audio in the moving image; subtitle generation means for generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating a portion of the second spoken sentence from the beginning to a middle of the second spoken sentence and a trailing partial subtitle indicating at least a portion of the second spoken sentence from the middle onwards; required time acquisition means for acquiring a required time for audibly outputting the leading partial subtitle; audio acquisition means for acquiring second audio in the second language corresponding to the first audio; and synthesis means for synthesizing the second audio, the leading partial subtitle, and the trailing partial subtitle with the moving image so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the trailing partial subtitle is output at a timing when the required time has elapsed since the output timing of the first audio.

[0007] (2) In the moving image generation system described in (1), the audio output timing acquisition means may input the moving image to a voice recognition engine, and may acquire the output timing of the first utterance sentence and the first voice based on a recognition result by the voice recognition engine.

[0008] (3) In the video generation system described in (2) above, the speech recognition engine may output, as the recognition result, a plurality of character strings each indicating a part of the first speech and output timings of the partial speech associated with the character strings. The speech output timing acquisition means may acquire the first utterance sentence by concatenating the plurality of character strings. The output timing of the partial speech associated with the first character string may be acquired as the output timing of the first speech.

[0009] (4) In the video production system described in (2) and (3) above, the subtitle production means may input the first utterance sentence to a translation engine, and may acquire the second utterance sentence generated by the translation engine.

[0010] (5) In the video generation system described in (4) above, the subtitle generation means may generate the first partial subtitle and the subsequent partial subtitle by dividing the second utterance sentence when the number of characters in the second utterance sentence is equal to or greater than a predetermined threshold.

[0011] (6) In the moving image generation system according to any one of (1) to (5), the voice acquisition means may input the second utterance sentence to a voice synthesis engine, and may acquire the second voice generated by the voice synthesis engine.

[0012] (7) In the moving image generation system according to any one of (1) to (6), the required time acquisition means may input the first partial subtitle to a speech synthesis engine, acquire audio representing the first partial subtitle generated by the speech synthesis engine, and acquire the length of the audio representing the first partial subtitle as the required time.

[0013] (8) A moving image generation method according to the present invention includes: a moving image acquisition step of acquiring moving image including first audio indicating a first spoken sentence in a first language; an audio output timing acquisition step of acquiring an output timing of the first audio in the moving image; a subtitle generation step of generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating a portion of the second spoken sentence from the beginning to a middle of the second spoken sentence and a trailing partial subtitle indicating at least a portion of the second spoken sentence from the middle onwards; a required time acquisition step of acquiring a required time for audio output of the leading partial subtitle; an audio acquisition step of acquiring second audio in the second language corresponding to the first audio; and a synthesis step of synthesizing the second audio, the leading partial subtitle, and the trailing partial subtitle with the moving image so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the trailing partial subtitle is output at a timing when the required time has elapsed since the output timing of the first audio.

[0014] (9) A video generation program according to the present invention operates a computer as a video acquisition means for acquiring a video including a first audio indicating a first spoken sentence in a first language, an audio output timing acquisition means for acquiring an output timing of the first audio in the video, a subtitle generation means for generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating from the beginning to a middle of the second spoken sentence and a trailing partial subtitle indicating at least a part of the second spoken sentence from the middle onwards, a required time acquisition means for acquiring a required time for audio output of the leading partial subtitle, an audio acquisition means for acquiring a second audio in the second language corresponding to the first audio, and a synthesis means for synthesizing the second audio, the leading partial subtitle and the trailing partial subtitle with the video so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the trailing partial subtitle is output at a timing when the required time has elapsed since the output timing of the first audio.

[0015] FIG. 1 is an overall configuration diagram of a video production system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a screen of an original video and a translated video according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of a hardware configuration of a video production device according to an embodiment of the present invention. FIG. 4 is a functional block diagram of a video production device according to an embodiment of the present invention. FIG. 5 is a detailed functional block diagram of a voice recognition unit according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of voice recognition data. FIG. 7 is a detailed functional block diagram of a translation unit according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of pre-translation data. FIG. 9 is a diagram showing an example of translated data. FIG. 10 is a detailed functional block diagram of a voice synthesis unit according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of translated voice data. FIG. 12 is a detailed functional block diagram of a subtitle production unit according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of subtitle text data. FIG. 14 is a diagram showing an example of a screen on which a first partial subtitle is displayed and a screen on which a subsequent partial subtitle is displayed. FIG. 15 is a flow diagram showing an example of the flow of a subtitle time code setting process for a subsequent partial subtitle according to an embodiment of the present invention. FIG. 16 is a diagram showing an example of subtitle display data. FIG. 17 is a detailed functional block diagram of a video production unit according to an embodiment of the present invention. FIG. 18 is a diagram showing an example of a timing chart of subtitle display text.

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is a diagram showing an example of the overall configuration of a video generation system 1 according to this embodiment.

[0018] 1, the moving image production system 1 according to this embodiment includes a speech recognition engine 2, a translation engine 3, a speech synthesis engine 4, a user terminal 6, and a moving image production device 100. The speech recognition engine 2, the translation engine 3, the speech synthesis engine 4, the user terminal 6, and the moving image production device 100 are connected to a computer network 5 such as the Internet. The speech recognition engine 2, the translation engine 3, the speech synthesis engine 4, and the moving image production device 100 are capable of communicating with each other via the computer network 5.

[0019] The voice recognition engine 2 is, for example, a computer system such as a server computer that performs voice recognition on the English voice included in the original video 60. The voice recognition engine 2 may be configured from a single computer or multiple computers.

[0020] The translation engine 3 is a computer system such as a server computer that executes a process of translating text in a first language into text in a second language, for example, from English text to Japanese text. For example, in this embodiment, Japanese text, which is the first language, is translated into English text, which is the second language. The translation engine 3 may be composed of a single computer or multiple computers.

[0021] The speech synthesis engine 4 is, for example, a computer system such as a server computer that executes speech processing such as speech synthesis. The speech synthesis engine 4 may be configured from a single computer or multiple computers.

[0022] The speech recognition engine 2, the translation engine 3, and the speech synthesis engine 4 may be provided by the same business operator, or may be provided by different business operators.

[0023] The user terminal 6 according to this embodiment is a computer used by a user. For example, the user terminal 6 is a smartphone, a tablet terminal, or a personal computer.

[0024] The video production device 100 according to this embodiment is a computer system such as a server computer that adds translated subtitles and translated audio obtained by translating the audio of an original video input by a user to the original video. Note that the video production device 100 may be configured from a single computer or multiple computers.

[0025] A user of the video creation system 1 according to this embodiment uploads an original video 60 to the video creation device 100 from a user terminal 6 via a computer network 5. This original video 60 includes, as an example, a plurality of sounds in English (English audio). That is, when the original video 60 is played, the English audio is output in sync with the video. When uploading the original video 60 to the video creation device 100, the user selects English, which is the original language included in the original video 60. At the same time, the user also selects Japanese as the translation target language for translating the subtitles and audio to be added to the original video 60.

[0026] 2A and 2B are diagrams illustrating an example of the screen of an original video 60 and a translated video 62 according to an embodiment of the present invention. FIG. 2A shows an example of the screen of an original video 60 uploaded by a user. The upper part of FIG. 2A shows the screen of the original video 60, and the lower part of FIG. 2A shows the output English audio. The original video 60 uploaded by the user includes English audio, and the English audio is played at a specific timing within the original video 60. Next, FIG. 2B shows an example of the screen of a translated video 62 generated by the video generation system 1. The upper part of FIG. 2B shows the screen of the translated video 62 with Japanese subtitles displayed in the subtitle display area 64, and the lower part of FIG. 2B shows the English audio and Japanese audio being played. As shown in FIG. 2B, the translated video 62 generated by the video generation system 1 according to this embodiment displays Japanese subtitles and plays the Japanese audio at the same time as the English audio is played.

[0027] The English audio, Japanese subtitles, and Japanese audio in the translated video 62 all have the same meaning. The Japanese subtitles are a Japanese translation of the content of the English audio, and are displayed in the subtitle display area 64 within the translated video 62. The Japanese audio is a synthesized audio of text translated by processing the results of recognizing the English audio. In this way, the user can obtain the translated video 62 to which Japanese subtitles and Japanese audio have been added, representing the content indicated by the English audio of the original video 60.

[0028] The hardware configuration for realizing the above-described moving image generating device 100 will be described below.

[0029] Fig. 3 is a diagram showing an example of the hardware configuration of a video production device 100 according to an embodiment of the present invention. The video production device 100 according to this embodiment is a computer as shown in Fig. 3. Software installed on the computer realizes a function of adding Japanese subtitles and Japanese audio that are translated from the English audio of an input original video 60. As shown in Fig. 3, an example of the video production device 100 according to the embodiment includes, for example, a processor 100a, a storage unit 100b, and a communication unit 100c.

[0030] The processor 100a is, for example, a program-controlled device such as a microprocessor that operates according to a program installed in the video generation device 100. The storage unit 100b is, for example, a storage element such as a ROM or RAM, a solid-state drive, or a hard disk drive. The storage unit 100b stores programs to be executed by the processor 100a. The communication unit 100c is, for example, a communication interface for exchanging data with the user terminal 6 via the computer network 5.

[0031] 4 is a functional block diagram of a video production system 1 according to an embodiment of the present invention. As shown in FIG. 4, the video production system 1 functionally includes a speech recognition unit 10, a translation unit 20, a speech acquisition unit 30, a subtitle generation unit 40, and a video production unit 50.

[0032] The above functions are implemented by executing a program containing instructions corresponding to the above functions on the processor 100a, which is installed in the computer, moving image production device 100. This program is supplied to the moving image production device 100 via a computer-readable information storage medium, such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, or flash memory, or via the Internet, for example.

[0033] The following describes a function for generating a translated video 62 with Japanese subtitles and Japanese audio added. In this embodiment, the original video 60 is assumed to contain three English audio segments.

[0034] First, a description will be given of the speech recognition unit 10. The speech recognition unit 10 recognizes the English speech of the original video 60 and generates speech-recognized text.

[0035] 5 is a detailed functional block diagram of the voice recognition unit 10. The voice recognition unit 10 includes a video acquisition unit 12 and a voice output timing acquisition unit 14. The video acquisition unit 12 and the voice output timing acquisition unit 14 are implemented mainly by a processor 100a and a communication unit 100c.

[0036] The video acquisition unit 12 acquires the original video 60 uploaded by the user and the settings of the original language and the translation target language selected by the user. In this embodiment, the settings acquired are English as the original language and Japanese as the translation target language. The video acquisition unit 12 may also store other user information and options selected by the user. For example, settings such as whether or not to include Japanese subtitles in the background and whether to synthesize Japanese audio using a male or female voice are included.

[0037] The audio output timing acquisition unit 14 transmits the audio included in the original video 60 acquired by the video acquisition unit 12 to the voice recognition engine 2 for voice recognition. The video acquisition unit 12 may input the original video 60 to the voice recognition engine 2.

[0038] The speech recognition engine 2 recognizes the speech input from the speech output timing acquisition unit 14 and outputs speech recognition text 112, which is the result of speech recognition of the English speech. Since the speech of the input original video 60 includes multiple English speeches, multiple speech recognition texts are output. Here, English speech refers to a single sentence spoken in English. The speech recognition text 112 may be output as a single sentence, just like the English speech, or, if there is a silent section in the middle of the English speech, multiple speech recognition texts 112 may be output as a character string indicating part of the English speech.

[0039] Furthermore, during speech recognition, the speech recognition engine 2 acquires speech time codes 110 in the original video 60 for the speech associated with each speech recognition text 112. The speech time codes 110 include a speech start time code 110a, which is the timing at which the English speech included in the original video 60 is output, and a speech end time code 110b, which is the timing at which the speech of the English speech ends.

[0040] Then, the voice recognition engine 2 transmits voice recognition data Dvr, which is data in which the voice recognition text 112, the speech start time code 110a, and the speech end time code 110b are associated, to the voice output timing acquisition unit 14. The voice output timing acquisition unit 14 transmits the voice recognition data Dvr to the translation unit 20.

[0041] Fig. 6 is a diagram showing an example of the voice recognition data Dvr. Fig. 6 shows the results of the voice recognition engine 2 recognizing the English voice of the original video 60. The recognition ID is an identifier assigned to each voice recognition result output by the voice recognition engine 2.

[0042] As shown in FIG. 6 , the speech recognition text 112-1 with a recognition ID of "1" indicates that the speech was spoken between "00:00.066" and "00:01.950" in the original video 60. The speech recognition text 112-2 with a recognition ID of "2" indicates that the speech was spoken between "00:02.586" and "00:3.450" in the original video 60. The speech recognition text 112-3 with a recognition ID of "3" indicates that the speech was spoken between "00:03.750" and "00:6.660" in the original video 60. The speech recognition text 112-4 with a recognition ID of "4" indicates that the speech was spoken between "00:06.690" and "00:8.310" in the original video 60.

[0043] Here, the speech recognition texts 112-1 and 112-4 are obtained by speech recognition of the entire English speech of one sentence contained in the original video 60. On the other hand, the speech recognition texts 112-2 and 112-3 are obtained by speech recognition of what should originally be a single sentence of English speech, splitting it into two character strings representing parts of the English speech. In this embodiment, the original video 60 contains three English speeches, but the speech recognition engine 2 outputs four speech recognition texts 112 as shown in FIG. 6 .

[0044] Next, the translation unit 20 in Fig. 4 will be described. The translation unit 20 generates an English utterance sentence representing an English voice from the speech-recognized text acquired from the speech recognition unit 10. The translation unit 20 then has the translation engine 3 translate the English utterance sentence to generate a Japanese utterance sentence 124.

[0045] Figure 7 is a detailed functional block diagram of the translation unit 20. As shown in Figure 7, the translation unit 20 includes a translation text generation unit 22 and a translation text storage unit 24. The translation text generation unit 22 includes a translation text processing unit 22a and an output timing adjustment unit 22b. The translation text generation unit 22 is implemented mainly using a processor 100a and a communication unit 100c. The translation text storage unit 24 is implemented mainly using a storage unit 100b.

[0046] The translated text generation unit 22 receives the speech recognition data Dvr from the speech output timing acquisition unit 14. Rather than translating the speech recognition text 112, the translated text generation unit 22 generates an English utterance sentence 122 that indicates the entire English speech sentence, and then has the translation engine 3 translate it. Having the translation engine 3 translate the English utterance sentence 122, which is a complete sentence, rather than fragmented speech recognition text 112, can improve translation accuracy.

[0047] The translation text processing unit 22a included in the translation text generation unit 22 generates an English utterance sentence 122 based on the speech-recognition text 112. If the speech-recognition text 112 represents an entire sentence of English speech, the translation text processing unit 22a acquires the speech-recognition text 112 as the English utterance sentence 122. If the speech-recognition text 112 is a character string that is less than a sentence and is part of English speech, the translation text processing unit 22a concatenates multiple speech-recognition texts 112 to acquire one English utterance sentence 122.

[0048] Next, the output timing adjustment unit 22b included in the translation text generation unit 22 generates a translation time code 120 for the English utterance sentence 122. If the speech-recognition text 112 represents the entire English speech, the speech time code 110 of the speech-recognition text 112 is set as the translation time code 120. If the speech-recognition text 112 is a character string that is less than a full sentence and is part of English speech, the speech start time code 110a of the first speech-recognition text 112 of the multiple speech-recognition texts 112 to be concatenated is set as the translation start time code 120a, and the speech end time code 110b of the last speech-recognition text 112 of the multiple speech-recognition texts 112 to be concatenated is set as the translation end time code 120b.

[0049] The translated text generation unit 22 may determine whether the speech-recognition text 112 represents the entire English speech, for example, based on whether a symbol indicating the end of a sentence (here, a period) is detected from the speech-recognition text 112. For example, when there is a speech-recognition text 112-m with a recognition ID of "m" in which no period is detected and a speech-recognition text 112-m+1 with a recognition ID of "m+1" in which a period is detected, the speech-recognition text 112-m and the speech-recognition text 112-m+1 may be concatenated to generate a single English utterance sentence 122. Alternatively, the English utterance sentence 122 may be generated from multiple speech-recognition texts 112 using a syntactic analysis method. Alternatively, the English utterance sentence 122 may be generated using a machine learning model that is trained to input multiple speech-recognition texts 112 and output multiple English utterance sentences 122 representing complete sentences.

[0050] Fig. 8 is a diagram showing an example of the pre-translation data Det. Fig. 8 shows the pre-translation data Det that is the result of the translation text generation unit 22 processing the speech-recognition text 112 before having the translation engine 3 translate it. The text ID is an identifier assigned to each English utterance sentence 122, which is one sentence of the speech-recognition text 112. The translation time code 120 includes a translation start time code 120a that indicates the output timing of the English utterance sentence 122, and a translation end time code 120b that indicates the end timing.

[0051] For a speech-recognition text 112-1 (see FIG. 6) that represents an entire sentence of English speech, the translation text processing unit 22a acquires the speech-recognition text 112-1 as is as an English utterance sentence 122-1. At this time, the translation text processing unit 22a sets the text ID of the English utterance sentence 122-1 to "1."

[0052] Next, the translated text processing unit 22a concatenates the speech-recognition text 112-2 and the speech-recognition text 112-3 (see FIG. 6), which are output as character strings less than a sentence that are part of the English speech, to obtain the English utterance sentence 122-2. At this time, the translated text processing unit 22a sets the text ID of the English utterance sentence 122-1 to "2."

[0053] Similarly to the speech-recognition text 112-1, the speech-recognition text 112-4 (see FIG. 6) representing the entire English speech of one sentence is also acquired as the English spoken sentence 122-3, and the text ID is set to "3."

[0054] In this way, the speech recognition result by the speech recognition engine 2 can be acquired as an English spoken sentence 122 representing one sentence.

[0055] Next, for an English utterance sentence 122-1 corresponding to a speech-recognition text 112-1 representing an entire sentence of English speech, the output timing adjustment unit 22b sets the speech time code 110-1 of the speech-recognition text 112-1 to the translation time code 120-1 of the English utterance sentence 122-1. That is, the translation start time code 120a-1 of the English utterance sentence 122-1 is set to "00:00.066" and the translation end time code 120b-1 is set to "00:01.950."

[0056] Here, the English utterance sentence 122-2 is obtained by concatenating the speech-recognition text 112-1 at the beginning and the speech-recognition text 112-3 at the end. The output timing adjustment unit 22b sets the translation start time code 120a-2 of the English utterance sentence 122-2 to the speech start time code 110a-2 of the speech-recognition text 112-2, and sets the translation end time code 120b-2 to the speech end time code 110b-3 of the speech-recognition text 112-3. In other words, the translation start time code 120a-2 of the English utterance sentence 122-2 is set to "00:02.586," and the translation end time code 120b-2 is set to "00:06.660."

[0057] Then, in the same way as setting the translation time code 120-1 for the English utterance sentence 122-1, the translation time code 120-3 for the English utterance sentence 122-3 is set to the utterance time code 110-4 of the speech-recognition text 112-4 corresponding to the English utterance sentence 122-3. That is, the translation start time code 120a-3 for the English utterance sentence 122-3 is set to "00:06.690", and the translation end time code 120b-3 for the English utterance sentence 122-3 is set to "00:08.310".

[0058] In this way, even if the speech recognition engine 2 does not recognize a single sentence as a result of speech recognition, it is possible to associate the translation time code 120 with each English utterance sentence 122 .

[0059] Next, the translated text generation unit 22 inputs the English utterance sentence 122 into the translation engine 3, and obtains a Japanese utterance sentence 124 obtained by translating the English utterance sentence 122 into Japanese. The Japanese utterance sentence 124 corresponds to the English utterance sentence 122, and therefore shares a text ID with the English utterance sentence 122. Because the English utterance sentence 122 and the Japanese utterance sentence 124 express the same meaning in different languages, it is preferable that the output timing of the Japanese utterance sentence 124 be the same as that of the English utterance sentence 122. The Japanese utterance sentence 124 shares the translation time code 120 of the corresponding English utterance sentence 122.

[0060] The translated text generating unit 22 then transmits the translated data Djt, which is data associating the English utterance sentence 122, the Japanese utterance sentence 124, and the translation time code 120, to the audio acquiring unit 30 and the subtitle generating unit 40.

[0061] The translated text storage unit 24 stores the translated data Djt.

[0062] 9 is a diagram showing an example of the translated data Djt. For the Japanese utterance sentence 124-1, the text ID "1" of the corresponding English utterance sentence 122-1 and the translation time code 120-1 are set. For the Japanese utterance sentence 124-2, the text ID "2" of the corresponding English utterance sentence 122-2 and the translation time code 120-2 are set. For the Japanese utterance sentence 124-3, the text ID "3" of the corresponding English utterance sentence 122-3 and the translation time code 120-3 are set.

[0063] Next, a description will be given of the speech acquisition unit 30 in Fig. 4. The speech acquisition unit 30 transmits the Japanese utterance sentence 124 generated by the translation unit 20 to the speech synthesis engine 4 to generate Japanese speech 130.

[0064] 10 is a detailed functional block diagram of the speech acquisition unit 30 according to an embodiment of the present invention. As shown in FIG. 10, the speech acquisition unit 30 includes, for example, a translated speech acquisition unit 32, a translated speech storage unit 34, and an integration speech generation unit 36 ​​as functions. The translated speech acquisition unit 32 and the integration speech generation unit 36 ​​are implemented primarily in the processor 100a and communication unit 100c. The translated speech storage unit 34 is implemented primarily in the storage unit 100b.

[0065] The translated speech acquisition unit 32 receives the translated data Djt from the translated text generation unit 22. The translated speech acquisition unit 32 transmits the Japanese utterance sentence 124 from the translated data Djt to the speech synthesis engine 4 for speech synthesis. The speech synthesis engine 4 transmits Japanese speech 130, which is generated by speech synthesis of the Japanese utterance sentence 124, to the translated speech synthesis unit. By speech synthesizing the Japanese utterance sentence 124 representing a single sentence, it is possible to generate smoother, more natural Japanese speech 130.

[0066] When the speech synthesis engine 4 synthesizes speech and generates the Japanese speech 130, silent intervals may be added to the beginning and end of the speech data. In this case, only the intervals where speech is detected may be extracted using a general speech interval detection technique and used as the Japanese speech 130.

[0067] Furthermore, when the Japanese speech 130 is longer than the corresponding English speech, it is preferable to perform processing to shorten the Japanese speech 130. Processing to shorten the Japanese speech 130 includes, for example, changing the playback speed of the Japanese speech 130 so that it fits within the length of the English speech.

[0068] It is preferable that the Japanese speech 130 be output at the same timing as the corresponding English speech. Here, the translation start time code 120a is set so that the output timing of the Japanese utterance sentence 124 when generating the Japanese speech 130 is the same as the output timing of the corresponding English utterance sentence 122 and the English speech. Therefore, the translation start time code of the Japanese utterance sentence 124 can be set for the Japanese speech 130 as well. Therefore, the Japanese speech 130 shares the translation time code 120 of the corresponding Japanese utterance sentence 124.

[0069] The translated speech acquisition unit 32 transmits to the translated speech storage unit 34 the translated speech data Djv, which is data that associates Japanese speech 130 with the translation time code 120 of the Japanese utterance sentence 124 that was transmitted to the speech synthesis engine 4 when generating the Japanese speech 130.

[0070] The translated speech storage unit 34 stores the translated speech data Djv.

[0071] 11 is a diagram showing an example of translated speech data Djv. Japanese speech 130-1 is set with the translation time code 120-1 of the corresponding Japanese utterance sentence 124-1. Japanese speech 130-2 is set with the translation time code 120-2 of the corresponding Japanese utterance sentence 124-2. Japanese speech 130-3 is set with the translation time code 120-3 of the corresponding Japanese utterance sentence 124-3.

[0072] Next, the integration speech generation unit 36 ​​generates integration speech 132 based on the translated speech data Djv stored in the translated speech storage unit 34. The integration speech generation unit 36 ​​synthesizes the multiple Japanese speeches into one integration speech 132 so that it is played back at the timing of each translation start time code 120a. The integration speech generation unit 36 ​​transmits the generated integration speech 132 to the speech synthesis unit 54.

[0073] In this embodiment, the integration audio 132 is generated so that Japanese audio 130-1 begins playback at translation start time code 120a-1, which is "00:00.066," Japanese audio 130-2 begins playback at translation start time code 120a-2, which is "00:02.586," and Japanese audio 130-3 begins playback at translation start time code 120a-3, which is "00:06.690."

[0074] Next, the subtitle generation unit 40 shown in Fig. 4 will be described. The subtitle generation unit 40 generates Japanese subtitles based on the Japanese utterance sentence 124 generated by the translation unit 20.

[0075] 12 is a detailed functional block diagram of the subtitle generation unit 40 according to an embodiment of the present invention. As shown in FIG. 12, the subtitle generation unit 40 includes, for example, a subtitle text generation unit 42 and a subtitle display text storage unit 44 as functions. The subtitle text generation unit 42 also includes a subtitle display text generation unit 42a, a subtitle time code generation unit 42b, and a required time acquisition unit 42c. The subtitle text generation unit 42 is implemented mainly using a processor 100a and a communication unit 100c. The subtitle display text storage unit 44 is implemented mainly using a storage unit 100b.

[0076] The subtitle text generation unit 42 generates subtitle display text 142 to be displayed on the translated video 62, and sets a subtitle time code 140 indicating the output timing of the subtitle display text 142. The subtitle text generation unit 42 also transmits subtitle display data Dst, in which the generated subtitle display text 142 and the subtitle time code 140 are associated, to the video generation unit 50.

[0077] The subtitle display text generation unit 42a included in the subtitle text generation unit 42 generates subtitle display text 142 to be displayed in the subtitle display area 64 in the translated video 62 based on each Japanese utterance sentence 124. The number of characters that can be displayed at one time is set in the subtitle display area 64. The subtitle display text may be displayed in the subtitle display area 64 with line breaks. Note that the font, character size, etc. may be set in the subtitle display area 64. Furthermore, when generating the subtitle display text 142, prohibition processing may be performed to prevent punctuation marks from appearing at the beginning of a sentence.

[0078] If the number of characters in the Japanese utterance sentence 124 is equal to or less than the number of characters that can be displayed at one time in the subtitle display area 64, the subtitle display text generation unit 42a sets the Japanese utterance sentence 124 as the subtitle display text 142. If the number of characters in the Japanese utterance sentence 124 is equal to or greater than the number of characters that can be displayed at one time in the subtitle display area 64, the subtitle display text generation unit 42a divides one Japanese utterance sentence 124 to generate multiple subtitle display texts 142.

[0079] Of the multiple subtitle display texts 142 generated when one Japanese utterance sentence 124 is divided, subtitle display text 142-n, which indicates the beginning to the middle of the Japanese utterance sentence 124, is called the first partial subtitle. Also, subtitle display text 142-n+1, ​​which indicates at least a part of the Japanese utterance sentence 124 from the middle onwards, is called the subsequent partial subtitle. When the Japanese utterance sentence 124 is divided into three parts, subtitle display text 142-n, which is the first partial subtitle, and subtitle display text 142-n+1 and subtitle display text 142-n+2, which are subsequent partial subtitles, are generated.

[0080] The subtitle display text generation unit 42a may divide the Japanese utterance sentence 124 into the first subtitle portion and the subsequent subtitle portion based on the number of characters or at a portion where the meaning of the Japanese utterance sentence 124 is clearly delimited.

[0081] 13 is a diagram showing an example of subtitle text data Dss. The subtitle text data Dss is data in which a text ID, a translation time code 120, subtitle display text 142, and a required time 146 are associated with each other. The subtitle display text 142 is generated for each text ID. A method for generating the subtitle display text 142 will be specifically described.

[0082] In this embodiment, it is assumed that the number of characters that can be displayed in the subtitle display area 64 is 32. For example, the number of characters in the Japanese utterance sentence 124-1 having the text ID of 1 is 17 (see FIG. 9), which is less than the number of characters that can be displayed in the subtitle display area 64. Therefore, the subtitle display text generation unit 42a sets the Japanese utterance sentence 124-1 as the subtitle display text 142-1.

[0083] Next, the number of characters in Japanese utterance sentence 124-2, whose text ID is "2," is "36" (see FIG. 9), which is greater than the number of characters that can be displayed in subtitle display area 64. Therefore, subtitle display text generation unit 42a divides Japanese utterance sentence 124-2 to generate subtitle display text 142-2 and subtitle display text 142-3. Here, subtitle display text 142-2 is the first partial subtitle, and subtitle display text 142-3 is the second partial subtitle.

[0084] 14A and 14B are diagrams showing examples of a screen on which a first partial subtitle is displayed and a screen on which a subsequent partial subtitle is displayed. In Fig. 14A, subtitle display text 142-2, which is the first partial subtitle, is displayed in the subtitle display area 64. In Fig. 14B, subtitle display text 142-3, which is the subsequent partial subtitle, is displayed in the subtitle display area 64.

[0085] Like the Japanese utterance sentence 124-1, the number of characters in the Japanese utterance sentence 124-3, whose text ID is "3," is "16" (see Figure 9), which is less than the number of characters that can be displayed in the subtitle display area 64, so the subtitle display text generation unit 42a sets the Japanese utterance sentence 124-3 as the subtitle display text 142-4.

[0086] Next, the subtitle time code 140 is set in the subtitle display text 142 generated by the subtitle display text generation unit 42a. The subtitle time code 140 includes a subtitle start time code 140a and a subtitle end time code 140b, which are output timings of the subtitle display text 142.

[0087] The subtitle time code generation unit 42b generates a subtitle time code 140 that indicates the timing at which the subtitle display text 142 is output. If the number of characters in the Japanese utterance sentence 124 is equal to or less than the number of characters that can be displayed in the subtitle display area 64, the subtitle time code generation unit 42b sets the translation time code 120 of the Japanese utterance sentence 124 to the subtitle time code 140 of the subtitle display text 142.

[0088] If the number of characters in the Japanese utterance sentence 124 is equal to or greater than the number of characters that can be displayed in the subtitle display area 64, the subtitle time code generation unit 42b sets the translation start time code 120a of the Japanese utterance sentence 124 to the subtitle start time code 140a-n of the subtitle display text 142-n, which is the first subtitle portion.

[0089] Here, subtitle display text 142-n+1, ​​which is the subsequent partial subtitle, does not have a translation start time code 120a for the corresponding Japanese spoken sentence 124. For subtitle display text 142-n+1, ​​which is the subsequent partial subtitle, the time required to audibly output subtitle display text 142-n, which is the corresponding first partial subtitle, is calculated and subtitle start time code 140a-n+1 is set.

[0090] The required time acquisition unit 42c transmits the subtitle display text 142-n, which is the first partial subtitle, to the speech synthesis engine 4, and generates a calculation speech 144-n. The required time acquisition unit 42c acquires a required time 146-n, which is the time required to audibly output the subtitle display text 142-n, which is the first partial subtitle, based on the calculation speech 144-n. When the speech synthesis engine 4 synthesizes speech and generates the calculation speech 144-n, silent intervals may be added to the beginning and end of the speech data. In such cases, the required time acquisition unit 42c may use the time of an interval in which speech is detected using a general speech interval detection technique as the required time 146-n.

[0091] The subtitle time code generator 42b then sets the subtitle start time code 140a-n of the subtitle display text 142-n, which is the leading subtitle portion, by adding the required time 146-n to the subtitle start time code 140a-n+1 of the subtitle display text 142-n+1, ​​which is the following partial subtitle. This allows the subtitle display text 142 to be switched at the appropriate timing even when there are multiple subtitle display texts 142 for one Japanese spoken sentence.

[0092] 15, an example of the flow of the subtitle time code setting process for the subsequent partial subtitles executed by the subtitle generation unit 40 will be described. It is assumed that the subtitle display text generation unit 42a divides the Japanese utterance sentence 124-n and generates the subtitle display text 142-n, which is the first partial subtitle, and the subtitle display text 142-n+1, ​​which is the subsequent partial subtitle.

[0093] In this processing example, the subtitle time code generation unit 42b sets the subtitle start time code 140a-n of the subtitle display text 142-n, which is the first subtitle portion, to the translation start time code 120a-n of the Japanese utterance sentence 124-n associated with the subtitle display text 142-n (S101).

[0094] The required time acquisition unit 42c transmits data of the subtitle display text 142-n, which is the first subtitle portion, to the speech synthesis engine 4 (S102).

[0095] Next, the speech synthesis engine 4 performs speech synthesis on the data of the subtitle display text 142-n received in the process shown in S102 to generate a speech for calculation 144-n (S201).

[0096] Then, the voice synthesis engine 4 transmits the calculation voice 144-n generated in the process shown in S201 to the required time acquisition unit 42c (S202).

[0097] The required time acquisition unit 42c receives the calculation-use voice 144-n transmitted in the process shown in S202 (S103).

[0098] Then, the required time acquisition unit 42c calculates the required time 146-n, which is the time required to audibly output the subtitle display text 142-n, which is the first subtitle portion, based on the calculation audio 144-n received in the process shown in S103 (S104).

[0099] Then, the subtitle time code generation unit 42b sets the subtitle start time code 140a-n of the subtitle display text 142-n, which is the subsequent partial subtitle, by adding the required time 146-n to the subtitle start time code 140a-n+1 of the subtitle display text 142-n+1 (S105).

[0100] By executing the process shown in S105, the subtitle time code setting process for the subsequent subtitle portion is completed.

[0101] The subtitle display text storage unit 44 stores the subtitle display data Dst.

[0102] 16 is a diagram showing an example of the subtitle display data Dst. The subtitle display data Dst is data in which a text ID, a subtitle time code 140, and a subtitle display text 142 are associated with each other.

[0103] First, the subtitle time code generator 42b sets the translation time code 120-1 of the corresponding Japanese utterance sentence 124-1 to the subtitle time code 140-1 of the subtitle display text 142-1. As shown in Fig. 16, the subtitle start time code 140a-1 of the subtitle display text 142-1 is set to "00:00.066" and the subtitle end time code 140b-1 is set to "00:01.95".

[0104] Next, the subtitle time code generator 42b sets the translation start time code 120a-2 of the Japanese utterance sentence 124-2 corresponding to the subtitle start time code 140a-2 of the subtitle display text 142-2, which is the first subtitle portion. As shown in Figure 16, the subtitle start time code 140a-1 of the subtitle display text 142-2 is set to "00:02.586".

[0105] The required time acquisition unit 42c then transmits the subtitle display text 142-2, which is the leading subtitle, to the speech synthesis engine 4, and generates the calculation speech 144-2. The required time 146-2 obtained by detecting the speech section of the calculation speech 144-2 is assumed to be 3.9 seconds (see FIG. 13).

[0106] Then, the subtitle start time code 140a-2 obtained by adding the required time of "3.9" seconds to the subtitle start time code 140a-3 of the subtitle display text 142-3, which is the subsequent partial subtitle, is set to "00:06.486".

[0107] In this way, the subtitle time code 140 can also be set for the subtitle display text 142 when the Japanese utterance sentence 124 is displayed in multiple parts.

[0108] The subtitle end time code 140b-2 of the subtitle display text 142-2 may be set to "00:06.486", the same as the subtitle start time code 140a-3 of the subtitle display text 142-3. By doing so, the subtitle display text 142-2 can be displayed until the subtitle display text 142-2 and the subtitle display text 142-3 are switched.

[0109] Furthermore, the translation end time code 120b-2 of the Japanese utterance sentence 124-2 with the text ID "2" may be set as the subtitle end time code 140b-3 of the subtitle display text 142-3.

[0110] Next, the subtitle time code generation unit 42b sets the translation time code 120-3 of the corresponding Japanese utterance sentence 124-3 to the subtitle time code 140-4 of the subtitle display text 142-4. As shown in Fig. 15, the subtitle start time code 140a-4 of the subtitle display text 142-3 is set to "00:06.690" and the subtitle end time code 140b-4 is set to "00:08.310".

[0111] Next, the video generation unit 50 in Fig. 4 will be described. The video generation unit 50 synthesizes the integration audio 132 generated by the audio acquisition unit 30 and the subtitle display text 142 generated by the subtitle generation unit 40 with the original video 60 to generate a translated video 62.

[0112] 17 is a detailed functional block diagram of the moving image generating unit 50 according to an embodiment of the present invention. As shown in FIG. 17, the moving image generating unit 50 includes, for example, functions such as a subtitle synthesis unit 52 and a voice synthesis unit 54. The subtitle synthesis unit 52 and the voice synthesis unit 54 are implemented mainly by the processor 100a and the communication unit 100c.

[0113] The subtitle synthesis unit 52 synthesizes the subtitle display data Dst transmitted from the subtitle generation unit 40 with the original moving image 60. The subtitle display data Dst is associated with a plurality of subtitle display texts 142 and subtitle start time codes 140a indicating the output timing of each of the subtitle display texts 142. The subtitle display data Dst can be synthesized with the original moving image 60 so that the subtitle display texts 142-n are output at the timing of the subtitle start time codes 140a-n.

[0114] The voice synthesis unit 54 synthesizes the integration voice 132 sent from the voice acquisition unit 30 with the original moving image 60. The integration voice 132 is synthesized so that the multiple Japanese voices 130 are output at the timing of their respective translation start time codes 120a, so the original moving image 60 and the integration voice 132 can be synthesized by matching their start timings.

[0115] The voice synthesis unit 54 may synthesize, instead of the integration voice 132, a plurality of Japanese voices 130-n into the original video 60 so that they are output in accordance with the associated translation time code 120-n.

[0116] The processing by the subtitle synthesis unit 52 and the processing by the voice synthesis unit 54 may be performed in any order.

[0117] Fig. 18 is a diagram showing an example of a timing chart of the subtitle display text 142. The horizontal axis in Fig. 18 represents the time axis, and indicates the timing at which the subtitle display text 142-2 and the subtitle display text 142-3 switch.

[0118] The subtitle synthesis unit 52 synthesizes the subtitle display text 142-2 with the original moving image 60 so that it is output at the timing of "00:02.586" set in the subtitle start time code 140a-2. The subtitle synthesis unit 52 also synthesizes the subtitle display text 142-3 with the original moving image 60 so that it is output at the timing of "00:06.486" set in the subtitle start time code 140a-3. When the audio synthesis unit 54 synthesizes the integration audio 132 with the original moving image 60, the Japanese audio 130-2 is played back at the timing of "00:02.586".

[0119] In this way, a Japanese utterance sentence 124-2 having the same meaning as the Japanese audio 130-2 is displayed at the timing of "00:02.586," and a translated video 62 can be generated in which the subtitle display text 142-2 and the subtitle display text 142-3 switch when the first subtitle portion of the Japanese utterance sentence 124-2 is read out.

[0120] As described above, the video creation system 1 of this embodiment allows a user to input an original video 60 containing English audio and generate a translated video with Japanese subtitles and Japanese audio added. In the translated video 62, Japanese audio 130 is played back at the timing of the English audio utterances, and Japanese subtitles are displayed. In particular, when a long sentence of Japanese subtitles is displayed multiple times as shown in Figure 17, the subtitles are switched in synchronization with the Japanese audio, reducing the risk of viewer confusion.

[0121] It should be noted that the present invention is not limited to the above-described embodiment. Here, the translation from English to Japanese has been described as an example, but the present invention can be similarly applied to translation between other languages. Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

Claims

1. A video generation system comprising: video acquisition means for acquiring video including a first audio indicating a first spoken sentence in a first language; audio output timing acquisition means for acquiring an output timing of the first audio in the video; subtitle generation means for generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating a portion from a beginning to a middle of the second spoken sentence, and a subsequent partial subtitle indicating at least a part of the second spoken sentence after the middle of the sentence; required time acquisition means for acquiring a required time for audio output of the leading partial subtitle; audio acquisition means for acquiring a second audio in the second language corresponding to the first audio; and synthesis means for synthesizing the second audio, the leading partial subtitle, and the subsequent partial subtitle with the video so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the subsequent partial subtitle is output at a timing when the required time has elapsed from the output timing of the first audio.

2. An animation generation system as described in claim 1, wherein the audio output timing acquisition means inputs the animation into a voice recognition engine and acquires the output timing of the first spoken sentence and the first audio based on the recognition result by the voice recognition engine.

3. An animation generation system as described in claim 2, wherein the voice recognition engine outputs a plurality of character strings indicating respective parts of the first voice and the output timing of the partial voices related to those character strings as the recognition result, and the voice output timing acquisition means acquires the first utterance sentence by concatenating the plurality of character strings and acquires the output timing of the partial voice related to the first character string as the output timing of the first voice.

4. A video generation system according to claim 2 or 3, wherein the subtitle generation means inputs the first spoken sentence into a translation engine and obtains the second spoken sentence generated by the translation engine.

5. A video generation system as described in claim 4, wherein the subtitle generation means generates the leading partial subtitle and the trailing partial subtitle by dividing the second utterance sentence when the number of characters in the second utterance sentence is equal to or greater than a predetermined threshold.

6. The video generation system according to claim 1, wherein the voice acquisition means inputs the second spoken sentence to a voice synthesis engine and acquires the second voice generated by the voice synthesis engine.

7. A moving image generation system according to claim 1, wherein said required time acquisition means inputs the first partial subtitle to a voice synthesis engine, acquires audio indicating the first partial subtitle generated by the voice synthesis engine, and acquires the length of the audio indicating the first partial subtitle as said required time.

8. A video generation method comprising: a video acquisition step of acquiring a video including a first audio indicating a first spoken sentence in a first language; an audio output timing acquisition step of acquiring an output timing of the first audio in the video; a subtitle generation step of generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating a portion from the beginning to a middle of the second spoken sentence, and a subsequent partial subtitle indicating at least a part of the second spoken sentence after the middle; a required time acquisition step of acquiring a required time for audio output of the leading partial subtitle; an audio acquisition step of acquiring a second audio in the second language corresponding to the first audio; and a synthesis step of synthesizing the second audio, the leading partial subtitle, and the subsequent partial subtitle with the video so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the subsequent partial subtitle is output at a timing when the required time has elapsed from the output timing of the first audio.

9. A program for causing a computer to operate as: a video acquisition means for acquiring a video including a first audio indicating a first spoken sentence in a first language; an audio output timing acquisition means for acquiring an output timing of the first audio in the video; a subtitle generation means for generating, based on a second spoken sentence in a second language corresponding to the first spoken sentence, a leading partial subtitle indicating from the beginning to a middle of the second spoken sentence, and a subsequent partial subtitle indicating at least a part of the second spoken sentence after the middle of the sentence; a required time acquisition means for acquiring a required time for audio output of the leading partial subtitle; an audio acquisition means for acquiring a second audio in the second language corresponding to the first audio; and a synthesis means for synthesizing the second audio, the leading partial subtitle, and the subsequent partial subtitle with the video so that the second audio is output at the output timing of the first audio, the leading partial subtitle is output at the output timing of the first audio, and the subsequent partial subtitle is output at a timing when the required time has elapsed from the output timing of the first audio.

Images