Information processing device, information processing method, and program

Abstract

This information processing device comprises: a speech section extraction unit that extracts a speech section from each of speech data in a first language added to video data and speech data in a second language after dubbing; a correspondence relation detection unit that detects the correspondence relation between the speech section of the first language and the speech section of the second language; and a position adjustment unit that adjusts, on the basis of the speech section of the first language, the position of the speech section of the second language in which the correspondence relation has been detected.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to a technology for dubbing audio for video.

[0002] Conventionally, in video content such as movies and dramas, the audio originally added to the video is dubbed into another language. For example, Patent Document 1 proposes a technique for correcting the video so that the mouth movements of the person being dubbed match those of the dubbing person who speaks the dubbing language.

[0003] Japanese Patent Application Publication No. 08-6182

[0004] The technology disclosed in Patent Document 1 requires complicated processing to correct the video to match the dubbed audio, and there is a risk that viewers may feel uncomfortable when watching the video. Therefore, there is a need for a further method for reducing the discrepancy between the dubbed audio and the video.

[0005] The present technology has been developed in view of the above circumstances, and aims to reduce the mismatch between audio and video after dubbing.

[0006] An information processing device according to the present technology includes a speech segment extraction unit that extracts speech segments from audio data in a first language added to video data and audio data in a second language after dubbing, a correspondence detection unit that detects a correspondence between the speech segments in the first language and the speech segments in the second language, and a position adjustment unit that adjusts the position of the speech segment in the second language for which the correspondence has been detected, based on the speech segment in the first language. According to the information processing device, by adjusting the position of the speech segment in the second language for which the correspondence has been detected, with respect to the speech segment in the first language, it is possible to more accurately align the position of the speech segment in the second language with the speech segment in the first language.

[0007] FIG. 1 is a diagram showing an overview of an information processing device. FIG. 2 is a diagram explaining audio extraction processing. FIG. 3 is a diagram explaining correspondence relationship detection processing. FIG. 4 is a diagram showing a specific example of correspondence relationship detection processing. FIG. 5 is a diagram explaining an example of correspondence relationship detection. FIG. 6 is a diagram explaining an example of position adjustment processing. FIG. 7 is a diagram showing an example of a method for calculating the degree of overlap. FIG. 8 is a flowchart showing the flow of audio position adjustment processing. FIG. 9 is a diagram explaining correspondence relationship detection processing in Modification 1. FIG. 10 is a diagram explaining correspondence relationship detection processing in Modification 2. FIG. 11 is a flowchart showing the flow of audio position adjustment processing in Modification 3. FIG. 12 is a flowchart showing the flow of audio position adjustment processing in Modification 4. FIG. 13 is a flowchart showing the flow of audio position adjustment processing in Modification 4.

[0008] Hereinafter, with reference to the accompanying drawings, an embodiment of a sensor device according to the present technology will be described in the following order: <1. Configuration of information processing device> <2. Audio position adjustment process> <3. Modification example> <4. Summary of embodiment> <5. Present technology>

[0009] 1 is a diagram showing an overview of an information processing device 1. The information processing device 1 is a computer device configured as, for example, a personal computer, a workstation, a mobile terminal device such as a smartphone or a tablet, or a server device in cloud computing.

[0010] The information processing device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, and a non-volatile memory unit 14 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory).

[0011] The CPU 11 executes various processes in accordance with programs stored in the ROM 12 or the non-volatile memory unit 14, or programs loaded from the storage unit 19 into the RAM 13. The RAM 13 also stores data necessary for the CPU 11 to execute various processes as appropriate.

[0012] The CPU 11 is provided with functional units related to the processes described below, including a speech segment extraction unit 31, a correspondence relationship detection unit 32, and a position adjustment unit 33. These functional units function based on software programs started by the CPU 11. These functional units will be described in detail later.

[0013] Instead of the CPU 11 or together with the CPU 11, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (Artificial Intelligence) processor, etc. may be provided.

[0014] The CPU 11, ROM 12, RAM 13, and nonvolatile memory unit 14 are interconnected via a bus 23. The input / output interface 15 is also connected to this bus 23.

[0015] An input unit 16 consisting of operators, operation devices, etc. is connected to the input / output interface 15. The input unit 16 may be various operators or operation devices, such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. The input unit 16 detects a user operation, and outputs a signal corresponding to the input operation to the CPU 11. A microphone may also be used as the input unit 16. Voice uttered by the user may also be input as operation information.

[0016] A display unit 17 and an audio output unit 18 are also connected to the input / output interface 15, either integrally or separately. The display unit 17 displays various information and is configured with an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, etc. The display unit 17 displays various images on the display screen based on instructions from the CPU 11. The display unit 17 also displays operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface). The audio output unit 18 outputs sound and is configured with a speaker, etc.

[0017] A storage unit 19 configured with a hard disk drive (HDD) or a solid state drive (SSD) and a communication unit 20 that communicates with other devices may be connected to the input / output interface 15. The storage unit 19 can store various types of data and programs.

[0018] The communication unit 20 performs communication processing via a transmission path such as the Internet, short-range wireless communication, wired communication with peripheral devices, bus communication, and the like.

[0019] Furthermore, a drive 21 is connected to the input / output interface 15 as required, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, memory card, or USB memory is appropriately attached.

[0020] The information processing device 1 is capable of acquiring video content through network communication via the communication unit 20. The information processing device 1 is also capable of reading video content recorded on a removable recording medium 22 using the drive 21. The acquired video content is stored in the storage unit 19, the video included in the video content is displayed on the display unit 17, and the audio included in the video content is output from the audio output unit 18.

[0021] Furthermore, in the information processing device 1, for example, a program for the processing of this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the program may be stored in advance in the ROM 12, the storage unit 19, or the like.

[0022] 2. Audio Position Adjustment Processing Next, the audio position adjustment processing executed by the CPU 11 of the information processing device 1 will be described.

[0023] Video content such as movies and dramas includes not only video data but also audio data such as lines spoken by actors. The audio data included in the video content is spoken in a specific language (for example, English).

[0024] When a specific language is dubbed into a language different from the specific language (e.g., Japanese), first, at the dubbing site (e.g., a studio), a dubber dubs lines in the dubbing language while watching the video, thereby generating audio data in the dubbing language. Then, the dubbed audio data is added to the video content, thereby generating video content with the dubbed audio data added. Note that the video content may include both the audio data before dubbing and the audio data after dubbing.

[0025] In this type of audio dubbing, the dubbing audio is collected while the video is being viewed, and since the characteristics of the original and post-dubbing languages ​​are different, there may be a slight discrepancy between the recorded audio and the video. This discrepancy between the video and audio creates a sense of discomfort for viewers watching the video content.

[0026] Therefore, the information processing device 1 adjusts the position of the dubbed audio to improve the accuracy of positioning when dubbing the audio. As a result, the information processing device 1 can reduce the discrepancy between the recorded dubbed audio and the video, thereby reducing the sense of discomfort felt by viewers watching the video content. Note that the position of the audio refers to the timing at which the audio is played back, i.e., its position in time.

[0027] In the following, an example will be described in which the audio (language) before dubbing added to video content is English and the audio (language) after dubbing is Japanese. Furthermore, the English audio will be referred to as English audio, and the Japanese audio will be referred to as Japanese audio.

[0028] First, the CPU 11 acquires video content including video data and English audio data, and dubbed Japanese audio data. The acquisition method may be from another device via the communication unit 20, or from the removable recording medium 22 via the drive 21. Note that the Japanese audio data may be one piece of data or may be divided into multiple pieces of data, since the dubber records the audio once or multiple times.

[0029] 2.1. Voice Extraction Process] Fig. 2 is a diagram illustrating the voice extraction process. The voice section extraction unit 31 executes voice extraction process to extract voice sections corresponding to sections in which actors or the like are speaking from the English voice data and Japanese voice data. In the voice extraction process, as shown in Fig. 2, a known voice extraction method is applied to voice waveforms 41 of the English voice data and Japanese voice data to extract speaking sections as voice sections 42.

[0030] Furthermore, the voice section extraction unit 31 extracts, from the English voice data and the Japanese voice data, a section sandwiched between adjacent voice sections 42 in which no speech is being made, as a silent section 43 .

[0031] In the following, the speech waveform 41 of the English speech data will be referred to as the English speech waveform 51, and the speech waveform 41 of the Japanese speech data will be referred to as the Japanese speech waveform 61. When describing the English speech waveform 51 and the Japanese speech waveform 61 without distinction, they will be referred to as the speech waveform 41. Furthermore, the speech interval 42 of the English speech data will be referred to as the English speech interval 52, and the speech interval 42 of the Japanese speech data will be referred to as the Japanese speech interval 62. When describing the English speech interval 52 and the Japanese speech interval 62 without distinction, they will be simply referred to as the speech interval 42. Furthermore, the silent interval 43 of the English speech data will be referred to as the English silent interval 53, and the silent interval 43 of the Japanese speech data will be referred to as the Japanese silent interval 63. When describing the English silent interval 53 and the silent interval 43 without distinction, they will be simply referred to as the silent interval 43.

[0032] [2.2. Correspondence Detection Process] Fig. 3 is a diagram illustrating the correspondence detection process. Fig. 3 illustrates a case where there are two pieces of Japanese speech data (referred to as Japanese speech 1 and Japanese speech 2, respectively). Fig. 3 also illustrates a volume waveform 44 and a speech waveform 41 for each piece of speech data. The volume waveform 44 of the English speech data is referred to as an English volume waveform 54, and the volume waveform 44 of the Japanese speech data is referred to as a Japanese volume waveform 64. When the English volume waveform 54 and the Japanese volume waveform 64 are described without distinction, they are referred to as volume waveform 44.

[0033] The correspondence detection unit 32 performs a correspondence detection process to detect the correspondence between the English speech section 52 and the Japanese speech section 62 extracted by the speech section extraction unit 31. Here, the correspondence refers to the relationship between the English speech section 52 containing an utterance in English and the Japanese speech section 62 containing an utterance in Japanese dubbed from that utterance. In other words, the correspondence refers to the relationship between the English speech section 52 and the Japanese speech section 62 containing Japanese speech dubbed from the English speech section in the English speech section 52.

[0034] In the example shown in Figure 3, the English speech section 52a is detected to correspond to Japanese speech sections 62a, 62b, and 62c of Japanese speech 1. Furthermore, the English speech section 52b is detected to correspond to Japanese speech section 62d of Japanese speech 1. Furthermore, the English speech section 52c is detected to correspond to Japanese speech section 62f of Japanese speech 2. Furthermore, the English speech section 52d is detected to correspond to Japanese speech section 62g of Japanese speech 2. Furthermore, the English speech section 52e is detected to correspond to Japanese speech section 62e of Japanese speech 1 and Japanese speech sections 62h and 62i of Japanese speech 2.

[0035] In this way, the correspondence between the English speech section 52 and the Japanese speech section 62 is not only a one-to-one correspondence, but also an N-to-M correspondence (N and M are integers equal to or greater than 1).

[0036] 4 shows a specific example of the correspondence detection process, in which an English volume waveform 54 and a Japanese volume waveform 64 are shown for explanation.

[0037] Here, since no dialogue is spoken in English or Japanese, the silent intervals 43 have almost the same timing and length. Therefore, in a specific example of the correspondence detection process, as shown in Figure 4, the correspondence detection unit 32 detects the correspondence between an English speech interval 52 and a Japanese speech interval 62 using the silent interval 43 as a reference.

[0038] First, the correspondence detection unit 32 detects the Japanese silent interval 63 that corresponds to the English silent interval 53 based on the degree of overlap (IoU: Intersection over Union) of the Japanese silent interval 63 with the English silent interval 53 .

[0039] Specifically, the correspondence detection unit 32 selects one English silent interval 53 as the detection target, and selects Japanese silent intervals 63 that exist within a predetermined range before and after the English silent interval 53. The correspondence detection unit 32 then calculates the degree of overlap between the English silent interval 53 being the detection target and the selected Japanese silent interval 63. Note that the correspondence refers to the relationship between the English silent interval 53 that includes a silent portion in English and the Japanese silent interval 63 that corresponds to that silent portion.

[0040] The overlap indicates the degree of correspondence between the position and width of the English silent interval 53 to be detected and the selected Japanese silent interval 63. If the positions and widths of the English silent interval 53 and the Japanese silent interval 63 match and completely overlap, the overlap is 1, and if the English silent interval 53 and the Japanese silent interval 63 do not overlap at all, the overlap is 0. Therefore, the greater the overlap, the stronger the correspondence between the English silent interval 53 to be detected and the selected Japanese silent interval 63. Note that the width of the silent interval 43 refers to the time length of the silent interval 43.

[0041] The correspondence detection unit 32 detects a Japanese silent interval 63 whose degree of overlap with the target English silent interval 53 is equal to or greater than a predetermined value (e.g., 0.5) as a Japanese silent interval 63 corresponding to the target English silent interval 53. The correspondence detection unit 32 detects corresponding Japanese silent intervals 63 for all English silent intervals 53.

[0042] In the example of Figure 4, a Japanese silent interval 63a corresponding to the English silent interval 53a is detected. A Japanese silent interval 63c corresponding to the English silent interval 53b is detected. A Japanese silent interval 63d corresponding to the English silent interval 53c is detected. A Japanese silent interval 63b located between the Japanese silent interval 63a and the Japanese silent interval 63c is determined not to correspond to any of the English silent intervals 53.

[0043] 5 shows an example of correspondence detection. When a Japanese silent interval 63 corresponding to an English silent interval 53 is detected, the correspondence detection unit 32 detects a corresponding combination of an English voice interval 52 existing between adjacent English silent intervals 53 and a Japanese voice interval 62 existing between Japanese silent intervals 63 corresponding to the English silent interval 53.

[0044] 5, an English voice interval 52f existing between the English silent intervals 53a and 53b and Japanese voice intervals 62j and 62k existing between the Japanese silent interval 63a and 63c are detected as a corresponding combination. Also, an English voice interval 52g existing between the English silent interval 53b and 53c and Japanese voice interval 62l existing between the Japanese silent interval 63c and 63d are detected as a corresponding combination.

[0045] In this way, the correspondence detection process uses the silent intervals 43 for English speech and Japanese speech as a reference to detect the correspondence between the English speech intervals 52 and the Japanese speech intervals 62. This makes it possible to accurately detect the correspondence even when the speech intervals 42 for English speech and Japanese speech have an N-to-M relationship, and also reduces the amount of calculation.

[0046] [2.3. Position Adjustment Processing] FIG. 6 is a diagram illustrating an example of the position adjustment processing.

[0047] When the correspondence detection unit 32 detects the correspondence between the English speech section 52 and the Japanese speech section 62, the position adjustment unit 33 executes a position adjustment process to adjust the position of the Japanese speech section 62 forward or backward with respect to the English speech section 52. Strictly speaking, adjusting the position of the Japanese speech section 62 refers to adjusting the position of the Japanese speech within the Japanese speech section 62.

[0048] For example, as shown in FIG. 6, when an English speech section 52f and Japanese speech sections 62j and 62k are detected as a corresponding combination, the position adjustment unit 33 adjusts the positions of the Japanese speech sections 62j and 62k based on the English speech section 52f.

[0049] Specifically, the range from the center of the English silent interval 53a that precedes the English speech interval 52f to the center of the English silent interval 53b that follows the English speech interval 52f is set as the position adjustment range within which the positions of the Japanese speech intervals 62j and 62k can be adjusted. This makes it possible to avoid overlap with adjacent Japanese speech intervals 62 when the position of the Japanese speech interval 62 is adjusted.

[0050] However, the position adjustment range is not limited to this, and may be, for example, between the start position of the English silent section 53a and the end position of the English silent section 53b, or may be only within the English speech section 52f.

[0051] Next, the position adjustment unit 33 shifts the Japanese voice sections 62j and 62k along the time axis at predetermined intervals within the position adjustment range, and calculates the degree of overlap (IoU) between the English volume waveform 54 of the English voice section 52a and the Japanese volume waveform 64 of the Japanese voice sections 62j and 62k at each position.

[0052] 7A and 7B are diagrams showing an example of a method for calculating the degree of overlap. To calculate the degree of overlap, the position adjustment unit 33 calculates the sum of the maximum values ​​by integrating the maximum values ​​of the English volume waveform 54 and the Japanese volume waveform 64 along the time axis, as shown in FIG. 7A. The position adjustment unit 33 also calculates the sum of the minimum values ​​by integrating the minimum values ​​of the English volume waveform 54 and the Japanese volume waveform 64 along the time axis, as shown in FIG. 7B. The position adjustment unit 33 then divides the sum of the minimum values ​​by the sum of the maximum values ​​to calculate the degree of overlap.

[0053] The overlap indicates the degree of match between the English volume waveform 54 and the Japanese volume waveform 64, and is 1 when the English volume waveform 54 and the Japanese volume waveform 64 are perfectly matched, and the value decreases as the English volume waveform 54 and the Japanese volume waveform 64 differ. Therefore, the greater the overlap, the more likely it is that the positions are aligned.

[0054] The position adjustment unit 33 calculates the degree of overlap for each of the Japanese speech segments 62j, 62k shifted at predetermined intervals within the position adjustment range. The position adjustment unit 33 then determines the position with the greatest degree of overlap as the position of the Japanese speech segments 62j, 62k. The position adjustment unit 33 then moves the Japanese speech segments 62j, 62k to the determined positions. The position adjustment unit 33 adjusts the position of the Japanese speech segments 62 for all corresponding combinations detected by the correspondence detection unit 32. The Japanese speech data whose position has been adjusted in this way is added to the video data to generate video content.

[0055] 8 is a flowchart showing the flow of the audio position adjustment process. When the audio position adjustment process starts, in step S1, the audio section extraction unit 31 executes audio section extraction processing to extract an English audio section 52 and a Japanese audio section 62 from the English audio data and the Japanese audio data, respectively. In step S1, the audio section extraction unit 31 also extracts an English silent section 53 and a Japanese silent section 63.

[0056] Next, in step S2, the correspondence detection unit 32 detects a Japanese silent interval 63 that corresponds to the English silent interval 53. Then, the correspondence detection unit 32 executes a correspondence detection process to detect the correspondence between the English speech interval 52 and the Japanese speech interval 62, using the corresponding English silent interval 53 and Japanese silent interval 63 as references.

[0057] In step S3, the position adjustment unit 33 executes a position adjustment process to adjust the position of the Japanese voice segment 62 in the forward and backward directions with respect to the English voice segment 52 as a reference.

[0058] 3. Modifications Note that the embodiment is not limited to the specific example described above, and various modification configurations are possible. For example, in the above embodiment, the first language before dubbing is English and the second language after dubbing is Japanese, but the first language and the second language are not limited to these and may be other languages.

[0059] 9 and 10 are diagrams illustrating the correspondence detection process in Modification 1. In the correspondence detection process in the above-described embodiment, the correspondence between the English speech interval 52 and the Japanese speech interval 62 sandwiched between the corresponding English silent interval 53 and Japanese silent interval 63 is detected as a reference. However, the method for detecting the correspondence between the English speech interval 52 and the Japanese speech interval 62 is not limited to this. For example, as shown in FIG. 9 , the correspondence detection unit 32 uses a hierarchical clustering technique to generate a hierarchical cluster 71 for the extracted English speech interval 52 by grouping the closest English speech intervals 52 or clusters into one cluster. In the example of FIG. 9 , it is assumed that three clusters 71a, 71b, and 71c are generated for the English speech data. Similarly, the correspondence detection unit 32 uses a hierarchical clustering technique to generate a hierarchical cluster 72 for the extracted Japanese speech interval 62 by grouping the closest Japanese speech intervals 62 or clusters into one cluster. In the example of FIG. 9, it is assumed that five clusters 72a, 72b, 72c, 72d, and 72e are generated for Japanese speech data.

[0060] After generating the clusters in this manner, the correspondence detection unit 32 calculates the degree of overlap (IoU) between the cluster 71 of the English speech data and the cluster 72 of the Japanese speech data, as shown in Fig. 10. The correspondence detection unit 32 then detects the correspondence between the English speech section 52 and the Japanese speech section 62 included in the cluster with the highest degree of overlap (cluster 71b and cluster 72d in the example of Fig. 9). In this way, the correspondence detection unit 32 can accurately detect the correspondence by clustering speech sections 42 of different lengths.

[0061] 11 is a diagram illustrating the correspondence detection process in Modification 2. As another example of a method for detecting the correspondence between the English speech section 52 and the Japanese speech section 62, the correspondence detection unit 32 calculates the degree of overlap between each English speech section 52 and all Japanese speech sections 62, as shown in FIG.

[0062] Then, for each English speech segment 52, the correspondence detection unit 32 detects the Japanese speech segment 62 with the highest degree of overlap as the corresponding Japanese speech segment 62.

[0063] [3.3. Modification 3] Fig. 12 is a flowchart showing the flow of the audio position adjustment process in Modification 3. In the above-described embodiment, the audio section extraction unit 31 extracts the English audio section 52 directly from the English audio data. However, the English audio data may contain background sounds in addition to the voices (speech portions) of the actors. Therefore, in the audio position adjustment process of Modification 3, as shown in Fig. 12, in step S11, the audio section extraction unit 31 performs a sound source separation process to extract the human speech portions from the English audio data. Note that since Japanese audio data is collected in a studio or the like and therefore does not generally contain background sounds, the audio section extraction unit 31 does not need to perform audio separation processing on the Japanese audio data.

[0064] Next, in step S1, the speech section extraction unit 31 extracts an English speech section 52 from the English speech data from which background sounds have been removed by the sound source separation process, and extracts a Japanese speech section 62 from the Japanese speech data. Thereafter, as in the above-described embodiment, the CPU 11 performs a correspondence relationship detection process in step S2 and a position adjustment process in step S3.

[0065] 13 is a flowchart showing the flow of the audio position adjustment process in Modification 4. In the above-described embodiment, the position adjustment unit 33 adjusts the position of the Japanese audio section 62 based on the English audio section 52. However, if the movement of the actor's mouth can be detected in the video, the position of the Japanese audio section 62 may be adjusted in accordance with the movement of the actor's mouth.

[0066] 13 , in the audio position adjustment process of Modification 4, the CPU 11 functioning as a movement detection unit detects the movement of the actor's mouth from the video data in step S21. Next, as in the embodiment, the CPU 11 performs the audio section extraction process in step S1 and the correspondence detection process in step S2. After that, in step S22, the position adjustment unit 33 adjusts the position of the Japanese audio section 62 based on the movement of the actor's mouth or the English audio section 52.

[0067] For example, when an actor's mouth movement is detected, the position adjustment unit 33 adjusts the position of the Japanese voice section 62 (one or more) corresponding to the mouth movement so that, for example, the timing at which the mouth movement starts coincides with the start of the Japanese voice section 62. The position adjustment unit 33 may also adjust the position of the Japanese voice section 62 corresponding to the mouth movement so that, for example, the timing at which the mouth movement ends coincides with the end of the Japanese voice section 62. Note that, for Japanese voice sections 62 other than those corresponding to the mouth movement, the positions are adjusted based on the corresponding English voice section 52, as in the embodiment. This makes it possible to further reduce the discrepancy between the video and the dubbed audio.

[0068] 14 is a flowchart showing the flow of the audio position adjustment process in Modification 5. In the above-described embodiment, the position adjustment unit 33 adjusts the position of the Japanese audio section 62 based on the English audio section 52. However, if the speech of each actor can be detected in the video, the position of the Japanese audio section 62 may be adjusted to match the speech of each actor.

[0069] 14, in the audio position adjustment process of Modification 5, in step S31, the CPU 11 functioning as a speaker diarization processor executes speaker diarization processing using speaker diarization technology to detect the speech and timing of each actor (speaker) in the English audio data. Next, as in the embodiment, the CPU 11 executes speech section extraction processing in step S1 and correspondence detection processing in step S2. Then, in step S32, the position adjustment unit 33 adjusts the position of the Japanese audio section 62 based on the speech and timing of each actor detected in the speaker diarization processing.

[0070] This makes it possible to adjust the position of the Japanese speech section 62 in accordance with the speech of each actor and its timing, thereby further reducing the discrepancy between the video and the dubbed audio.

[0071] 4. Summary of the Embodiment As described above, the information processing device 1 according to the embodiment includes a speech section extraction unit 31 that extracts speech sections 42 (English speech sections 52 and Japanese speech sections 62) from audio data in a first language (English speech data) added to video data and audio data in a second language (Japanese speech data) after dubbing, respectively; a correspondence detection unit 32 that detects a correspondence between a speech section in the first language and a speech section in the second language; and a position adjustment unit 33 that adjusts the position of a speech section in the second language for which a correspondence is detected, based on the speech section in the first language. After detecting the correspondence between the English speech section 52 and the Japanese speech section 62, the information processing device 1 adjusts the position of the Japanese speech section 62 that corresponds to the English speech section 52. This enables the information processing device 1 to more accurately align the position of the Japanese speech section 62 with the English speech section 52. Thus, the information processing device 1 can reduce the misalignment between the audio and video after dubbing.

[0072] The correspondence detection unit 32 can detect an N-to-M (N is an integer equal to or greater than 1) correspondence between a speech section of a first language (an English speech section 52) and a speech section of a second language (a Japanese speech section 62). Due to differences in the characteristics of different languages, the number of speech sections 42 before and after dubbing may differ. Therefore, by detecting an N-to-M correspondence between the English speech section 52 and the Japanese speech section 62, the correspondence detection unit 32 can reduce erroneous detection of correspondence due to differences in the characteristics between different languages.

[0073] The correspondence detection unit 32 detects the correspondence between the speech intervals of the first language (English speech intervals 52) and the speech intervals of the second language (Japanese speech intervals 62) using the silent intervals 43 as a reference. Since no dialogue occurs in an interval in English or Japanese, the silent intervals 43 have almost the same timing and length. Therefore, by detecting the correspondence between the English speech intervals 52 and the Japanese speech intervals 62 using the silent intervals 43 (English silent intervals 53, Japanese silent intervals 63) as a reference, it is possible to reduce erroneous detection of the correspondence.

[0074] The correspondence detection unit 32 detects corresponding silent intervals in the first language and the second language based on the degree of overlap between the silent intervals in the first language (English silent intervals 53) and the silent intervals in the second language (Japanese silent intervals 63). As described above, because the silent intervals 43 occur at the same timing regardless of language, the correspondence detection unit 32 calculates the degree of overlap between the English silent intervals 53 and the Japanese silent intervals 63, and detects combinations with a high degree of overlap as corresponding English silent intervals 53 and Japanese silent intervals 63. This makes it possible to accurately detect the correspondence between the English silent intervals 53 and the Japanese silent intervals 63.

[0075] The correspondence detection unit 32 detects, as a corresponding combination, a first language speech interval (English speech interval 52) existing between adjacent first language silent intervals (English silent intervals 53), and a second language speech interval (Japanese speech interval 62) existing between second language silent intervals (Japanese silent intervals 63) that correspond to the adjacent first language silent intervals. As described above, the correspondence between the English silent intervals 53 and the Japanese silent intervals 63 is detected with high accuracy. Therefore, the correspondence detection unit 32 can accurately detect the correspondence between the English silent interval 52 and the Japanese silent interval 62 that are sandwiched between the corresponding English silent intervals 53 and Japanese silent intervals 63.

[0076] The position adjustment unit 33 adjusts the position of the second language voice section (Japanese voice section 62) based on the volume waveforms 44 (English volume waveform 54, Japanese volume waveform 64) of the voice sections 42 (English voice section 52, Japanese voice section 62) of the first and second languages ​​for which a correspondence has been detected. The voice waveforms 41 may differ significantly between different languages ​​due to differences in intonation, subject-predicate position, etc. On the other hand, the volume is often the same before and after dubbing to match emotions, etc., even if the languages ​​are different. Therefore, by adjusting the position of the Japanese voice section 62 based on the volume waveform 44, the position adjustment unit 33 can absorb the differences in the voice waveforms 41 between different languages ​​and perform accurate position adjustment.

[0077] The position adjustment unit 33 adjusts the position based on the degree of overlap of the volume waveforms 44 (English volume waveform 54, Japanese volume waveform 64) of the speech segments of the first language and the second language, thereby enabling the position adjustment unit 33 to align the position of the Japanese speech to the position where the English volume waveform 54 and the Japanese volume waveform 64 are closest.

[0078] The position adjustment unit 33 moves the volume waveform of the voice section of the second language (Japanese volume waveform 64) at predetermined intervals relative to the volume waveform of the voice section of the first language (English volume waveform 54), and moves the voice section of the second language to a position where the overlap between the volume waveforms of the voice sections of the first and second languages ​​is maximized. This allows the position adjustment unit 33 to move the Japanese voice to an optimal position.

[0079] The position adjustment unit 33 moves the volume waveform of the speech section of the second language (Japanese volume waveform 64) within a position where a silent section (English silent section 53) adjacent to the corresponding speech section of the first language exists, thereby reducing overlap between Japanese speech sections caused by moving the Japanese speech sections.

[0080] The correspondence detection unit 32 hierarchically clusters the speech segments of the first language and the second language, and detects the correspondence between the speech segments of the first language and the speech segments of the second language based on the degree of overlap of the clustered speech segments. In this way, the correspondence detection unit 32 can accurately detect the correspondence by clustering speech segments of different lengths.

[0081] The voice section extraction unit 31 performs a voice separation process to extract human voice sections from the voice data in the first language (English voice data). This makes it possible to accurately extract the English voice section 52, which is the spoken section, even from voice data that includes background sounds.

[0082] The information processing method extracts speech intervals from audio data in a first language added to video data and audio data in a second language after dubbing, detects a correspondence between the speech intervals in the first language and the speech intervals in the second language, and adjusts the position of the speech interval in the second language for which a correspondence is detected, based on the speech interval in the first language. The program causes the information processing device 1 to execute a speech interval extraction process for extracting speech intervals from the audio data in the first language added to video data and the audio data in the second language after dubbing, a correspondence detection process for detecting a correspondence between the speech intervals in the first language and the speech intervals in the second language, and a position adjustment process for adjusting the position of the speech interval in the second language for which a correspondence is detected, based on the speech interval in the first language. This program realizes the functions of the information processing device described above on a computer. Furthermore, a recording medium having the program of the present invention recorded thereon facilitates providing the program, enabling the present invention to be widely available. This program can be pre-recorded on a recording medium such as a HDD built into a computer or a ROM in a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto-optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. Such removable recording media can be provided as so-called packaged software. In addition to being installed on a computer or the like from a removable recording medium, the program can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0083] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0084] <5. The Present Technology> The present technology may also have the following configuration. (1) An information processing device comprising: a speech section extraction unit that extracts speech sections from audio data in a first language added to video data and audio data in a second language after dubbing, respectively; a correspondence detection unit that detects a correspondence between the speech sections of the first language and the speech sections of the second language; and a position adjustment unit that adjusts the position of the speech sections of the second language for which the correspondence has been detected, based on the speech sections of the first language. (2) The information processing device described in (1), wherein the correspondence detection unit is capable of detecting an N (N is an integer equal to or greater than 1) to M (M is an integer equal to or greater than 1) correspondence between the speech sections of the first language and the speech sections of the second language. (3) The information processing device described in (1) or (2), wherein the correspondence detection unit detects the correspondence between the speech sections of the first language and the speech sections of the second language based on a silent section. (4) The information processing device according to (3), wherein the correspondence detection unit detects a correspondence between a silent interval of the first language and a silent interval of the second language based on an overlapping state between the silent intervals of the first language and the silent intervals of the second language. (5) The information processing device according to (3) or (4), wherein the correspondence detection unit detects, as a corresponding combination, a voice interval of the first language existing between adjacent silent intervals of the first language and a voice interval of the second language existing between silent intervals of the second language that correspond to adjacent silent intervals of the first language. (6) The information processing device according to any of (1) to (5), wherein the position adjustment unit adjusts a position of the voice interval of the second language based on volume waveforms of the voice intervals of the first language and the second language for which a correspondence has been detected. (7) The information processing device according to (6), wherein the position adjustment unit adjusts a position of the voice interval of the second language based on an overlapping state between volume waveforms of the voice intervals of the first language and the second language. (8) The information processing device according to (7), wherein the position adjustment unit moves the volume waveform of the voice section of the second language relative to the volume waveform of the voice section of the first language at predetermined intervals, and moves the voice section of the second language to a position where the degree of overlap between the volume waveforms of the voice sections of the first language and the second language is maximized.(9) The information processing device according to (8), wherein the position adjustment unit moves the volume waveform of the voice interval of the second language within a position where the silent interval adjacent to the corresponding voice interval of the first language exists. (10) The information processing device according to any of (1) to (9), wherein the correspondence detection unit hierarchically clusters the voice intervals of the first language and the second language, and detects the correspondence between the voice interval of the first language and the voice interval of the second language based on an overlap of the clustered voice intervals. (11) The information processing device according to any of (1) to (10), wherein the voice interval extraction unit performs a voice separation process to extract a human voice portion from the voice data of the first language. (12) The information processing device according to (5), further comprising a movement detection unit that detects mouth movement from the video, and wherein the position adjustment unit adjusts the position of the voice interval of the second language in accordance with the mouth movement. (13) An information processing method for extracting speech intervals from audio data in a first language added to video data and audio data in a second language after dubbing, detecting a correspondence between the speech intervals in the first language and the speech intervals in the second language, and adjusting the position of the speech interval in the second language for which the correspondence has been detected, based on the speech interval in the first language. (14) A program for causing an information processing device to execute the following steps: a speech interval extraction process for extracting speech intervals from audio data in a first language added to video data and audio data in a second language after dubbing, a correspondence detection process for detecting a correspondence between the speech intervals in the first language and the speech intervals in the second language, and a position adjustment process for adjusting the position of the speech interval in the second language for which the correspondence has been detected, based on the speech interval in the first language.

[0085] REFERENCE SIGNS LIST 1 Information processing device 11 CPU 31 Voice section extraction unit 32 Correspondence detection unit 33 Position adjustment unit

Claims

1. An information processing device comprising: a speech section extraction unit which extracts speech sections from audio data in a first language added to video data and audio data in a second language after dubbing; a correspondence detection unit which detects a correspondence between the speech section of the first language and the speech section of the second language; and a position adjustment unit which adjusts the position of the speech section of the second language for which a correspondence has been detected, based on the speech section of the first language.

2. The information processing device according to claim 1, wherein the correspondence detection unit is capable of detecting the correspondence between the speech intervals of the first language and the speech intervals of the second language in an N (N is an integer of 1 or more) to M (M is an integer of 1 or more) relationship.

3. The information processing device according to claim 1, wherein the correspondence detection unit detects the correspondence between the speech intervals of the first language and the speech intervals of the second language based on silent intervals.

4. The information processing device according to claim 3, wherein the correspondence detection unit detects the correspondence between the silent intervals of the first language and the silent intervals of the second language based on the degree of overlap between the silent intervals of the first language and the silent intervals of the second language.

5. The information processing device according to claim 3, wherein the correspondence detection unit detects, as a corresponding combination, a speech section of the first language that exists between adjacent silent sections of the first language, and a speech section of the second language that exists between silent sections of the second language that correspond to adjacent silent sections of the first language, respectively.

6. The information processing device according to claim 1, wherein the position adjustment unit adjusts the position of the voice section of the second language based on volume waveforms of the voice sections of the first language and the second language for which a correspondence has been detected.

7. The information processing device according to claim 6, wherein the position adjustment unit adjusts the position of the voice section of the second language based on the degree of overlap of volume waveforms of the voice sections of the first language and the second language.

8. The information processing device according to claim 7, wherein the position adjustment unit moves the volume waveform of the voice section of the second language relative to the volume waveform of the voice section of the first language at predetermined intervals, and moves the voice section of the second language to a position where the overlap between the volume waveforms of the voice sections of the first language and the second language is maximized.

9. The information processing device according to claim 8, wherein the position adjustment unit moves the volume waveform of the voice interval of the second language within a range in which the silent interval adjacent to the corresponding voice interval of the first language exists.

10. The information processing device according to claim 1, wherein the correspondence detection unit hierarchically clusters the speech segments of the first language and the second language, and detects the correspondence between the speech segments of the first language and the speech segments of the second language based on the degree of overlap of the clustered speech segments.

11. The information processing device according to claim 1, wherein the voice section extraction unit performs a voice separation process to extract human voice portions from the voice data in the first language.

12. The information processing device according to claim 1, further comprising a movement detection unit that detects mouth movement from the video data, and the position adjustment unit adjusts the position of the speech segment of the second language in accordance with the mouth movement.

13. An information processing method comprising: extracting speech intervals from audio data in a first language added to video data and audio data in a second language after dubbing; detecting a correspondence between the speech intervals in the first language and the speech intervals in the second language; and adjusting the position of the speech interval in the second language for which a correspondence has been detected, based on the speech interval in the first language.

14. A program that causes an information processing device to execute the following steps: a speech interval extraction process that extracts speech intervals from audio data in a first language added to video data and from audio data in a second language after dubbing; a correspondence detection process that detects a correspondence between the speech intervals in the first language and the speech intervals in the second language; and a position adjustment process that adjusts the position of the speech interval in the second language for which a correspondence has been detected, based on the speech interval in the first language.

Images