Subtitle display device and subtitle display program

Abstract

To provide a subtitle display device and a subtitle display program that can reduce the time from utterance to display of subtitles by displaying subtitles at a stage where speech recognition of a part of subtitles to be displayed on a screen is fixed and updating subtitles each time new recognition results are fixed.SOLUTION: A subtitle display device includes: a speech recognition unit that performs speech recognition processing for input speech; a stock that stores recognition results whenever the recognition results determined by the speech recognition unit are fixed; a timer; and a subtitle generator that starts the timer at timing when the recognition results are stored at the top of the stock and generates and outputs subtitle data based on the recognition results stored every specified time based on the timer.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a subtitle display device and a subtitle display program that have a small delay from speech to subtitle display without increasing the data amount of subtitles.

Background Art

[0002] Conventionally, technologies for creating subtitles for televisions, movies, etc. using speech recognition technology have been developed. For example, Patent Document 1 proposes a technology for sequentially recognizing input speech, creating a recognition result that is texturized, and determining a display time for each sentence included in the recognition result based on the speech time of the speech. Also, Non-Patent Document 1 proposes a technology for a sequential two-pass decoder for early determination of recognition results by performing second-pass processing sequentially during speech input.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When adding subtitles to live television programs, we consider sending out the results of speech recognition as closed captions for broadcast without any correction. The number of highly skilled operators capable of subtitling live television programs is limited, and there was a need for technology that could add subtitles without requiring human intervention in order to expand subtitling services. In addition, for various reasons, it took time to finalize the subtitles for live television programs. As a result, the subtitles were generally displayed with a significant delay of several seconds or more compared to the spoken words, and sometimes unnatural subtitles appeared at points where the topic of the program changed. The subtitle display method using speech recognition described in Patent Document 1 assumed that speech recognition would output as a sequence of words. Furthermore, similar to current subtitle broadcasting, if all subtitles to be displayed on the screen (for example, 30 characters in 15 characters x 2 lines) are acquired before displaying, there is a problem of a large delay between speech and subtitle display. On the other hand, using a speech recognition model that outputs speech recognition results for each word, as described in Non-Patent Document 1, it was possible to reduce latency by displaying subtitles each time the speech recognition result was finalized. However, when sending as closed captions, there was a problem in that the amount of data increased because the subtitle information to be displayed had to be resent each time the subtitles were updated.

[0006] The present invention aims to provide a subtitle display device and a subtitle display program that can reduce the delay between speech and subtitle display by displaying subtitles when the speech recognition of a portion of the subtitles to be displayed on the screen has been finalized, and updating the subtitles each time a new recognition result is finalized. [Means for solving the problem]

[0007] (1) The subtitle display device of the present invention comprises: a speech recognition unit that performs speech recognition processing on input audio; a stock that stores the recognition result each time the recognition result recognized by the speech recognition unit is determined; a timer; and a subtitle generation unit that starts the timer when a recognition result is stored at the beginning of the stock, and generates and outputs subtitle data based on the recognition result stored at predetermined time intervals based on the timer.

[0008] According to (1) above, by displaying the subtitles when the speech recognition for some of the subtitles to be displayed on the screen has been finalized, and updating the subtitles each time a new recognition result is finalized, the delay between speaking and displaying the subtitles can be reduced.

[0009] (2) The subtitle display device described in (1) includes a display time calculation unit that calculates the display time of the subtitle data generated and output by the subtitle generation unit based on the recognition result of the speech recognition unit.

[0010] According to (2) above, subtitles can be displayed according to the speaker's speaking speed, and subtitles that are easy for the user of the display device to read can also be displayed.

[0011] (3) In the subtitle display device described in (1) or (2), when the speech recognition unit recognizes a silent section, the subtitle generation unit generates and outputs subtitle data based on the recognition result stored in the stock, regardless of the value of the timer, deletes all stored recognition results from the stock, and resets the timer.

[0012] According to (3) above, subtitles for an utterance can be displayed at the end of the utterance.

[0013] (4) In the subtitle display device described in (1) or (2), if the amount of data of the recognition results stored in the stock exceeds the amount of data that can be displayed on one screen, the subtitle generation unit generates and outputs subtitle data of a number of characters that can be displayed on one screen from the recognition results stored in the stock, regardless of the value of the timer, deletes the recognition results used in the output subtitle data from the stock, and resets the timer.

[0014] According to (4) above, by displaying subtitles when the number of characters that can be displayed on a single screen is reached, both the amount of data and the latency can be kept to a minimum.

[0015] (5) The subtitle display program of the present invention is a subtitle display program that causes a computer to operate as a subtitle display device, and comprises: speech recognition means that performs speech recognition processing on input speech; stock means that saves the recognition result each time the speech recognition result is confirmed; and subtitle generation means that starts a timer included in the subtitle display device at the timing when the recognition result is saved first, and generates and outputs subtitle data based on the recognition results stored at predetermined time intervals based on the timer.

[0016] According to (5) above, the same effect as in (1) can be achieved. [Effects of the Invention]

[0017] According to the present invention, the delay between speaking and displaying subtitles can be reduced by displaying subtitles when the speech recognition of a portion of the subtitles to be displayed on the screen has been finalized, and updating the subtitles each time a new recognition result is finalized. [Brief explanation of the drawing]

[0018] [Figure 1] This figure shows an example of the configuration of a subtitle output system according to one embodiment. [Figure 2A] This figure shows an example of the recognition result. [Figure 2B] This figure shows an example of the recognition result. [Figure 3] It is a diagram showing an example of the time-series change of the subtitle data successively displayed. [Figure 4] It is a flowchart for explaining the subtitle output process of the subtitle output device. [Figure 5] It is a flowchart for explaining the detailed processing content of the subtitle output process in the speech section shown in step S3 in FIG. 4. [Figure 6] It is a flowchart for explaining the detailed processing content of the subtitle output process in the silent section shown in step S4 in FIG. 4.

Mode for Carrying Out the Invention

[0019] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. <One Embodiment> FIG. 1 is a diagram showing an example of the configuration of a subtitle output system according to an embodiment. Here, the case of displaying subtitles on the video of a TV program is illustrated. Note that the present invention is not limited to TV programs and is also applicable to movies, Internet-distributed videos, and the like. As shown in FIG. 1, the subtitle output system 1 includes a subtitle output device 10 as a subtitle display device, an audio input device 20, and a video generation device 30. The subtitle output device 10, the audio input device 20, and the video generation device 30 may be directly connected by wire or wirelessly, or may be connected via a network (not shown) such as a LAN (Local Area Network) or the Internet.

[0020] The audio input device 20 inputs audio to the subtitle output device 10. For example, the audio input device 20 may input an audio file of a TV program as audio to the subtitle output device 10. Also, for example, the audio input device 20 may be used as a microphone to input the audio input from the microphone to the subtitle output device 10.

[0021] The video generation device 30 inputs, for example, the subtitle data output by the subtitle output device 10 and outputs a broadcast closed caption.

[0022] The subtitle output device 10 is, for example, a computer, which performs speech recognition processing on the audio input from the audio input device 20, generates subtitle data based on the recognition results, and outputs it to the video generation device 30. As shown in Figure 1, the subtitle output device 10 includes a control unit 100, a storage unit 200, a stock 250, and a timer 300. The control unit 100 also includes an audio acquisition unit 110, an audio recognition unit 120, a subtitle generation unit 130, and a display time calculation unit 140.

[0023] The storage unit 200 is a memory device or an HDD (Hard Disk Drive), etc. The storage unit 200 stores the operating system executed by the control unit 100, which will be described later, and application programs such as a subtitle display program. Furthermore, the memory unit 200 may also store words included in the recognition results of the speech recognition unit 120, which will be described later, as well as the time of utterance of those words. The recognition results stored in the memory unit 200 will be described later.

[0024] Stock 250 consists of memory, buffers, etc., and stores the text consisting of the recognized words to be displayed as subtitles each time the recognition result is confirmed. Although Stock 250 has a different configuration from Storage Unit 200, it may be included in Storage Unit 200.

[0025] Timer 300 is a known timer that measures elapsed time based on control instructions from the control unit 100, which will be described later.

[0026] The control unit 100 includes a CPU, ROM, RAM, CMOS memory, etc., which are configured to communicate with each other via a bus, and is known to those skilled in the art. The CPU is a processor that controls the subtitle output device 10 as a whole. The CPU reads the system program and application programs such as the subtitle display program stored in ROM via the bus, and controls the entire subtitle output device 10 according to the system program and application programs. As a result, as shown in Figure 1, the control unit 100 is configured to realize the functions of the audio acquisition unit 110, the audio recognition unit 120, the subtitle generation unit 130, and the display time calculation unit 140. Various data such as temporary calculation data and display data are stored in the RAM. The CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that retains its memory state even when the power to the subtitle output device 10 is turned off.

[0027] The voice acquisition unit 110 acquires audio from the audio input device 20. The voice acquisition unit 110 outputs the acquired audio to the voice recognition unit 120, which will be described later.

[0028] The speech recognition unit 120 performs speech recognition processing on the speech input from the speech acquisition unit 110, for example, using known methods such as those described in Patent Document 1 and Non-Patent Document 1. The speech recognition unit 120 stores the recognition results in the storage unit 200. In addition, each time a recognition result is confirmed, the speech recognition unit 120 adds the recognized word to the stock 250. Figures 2A and 2B show examples of recognition results. For example, Figures 2A and 2B show the recognition result, including the "recognition result," "time of speech," and "time of recognition confirmation," when the speaker says, "The current temperature is 20.2 degrees. The daytime high is expected to rise to around 28 degrees, making it as hot as early July during the day." The "recognition results" store words (for example, "present" or "of") in the order they were recognized (spoken) by the speech recognition unit 120. The "Speech Time" column stores the time when the word in the "Recognition Result" was spoken. The "Recognition Confirmation Time" stores the time when the speech recognition unit 120 confirmed the recognition of the word in the "Recognition Result". Figures 2A and 2B also show data comparing the "display time" and "delay" when subtitle data is displayed sequentially with the "display time" and "delay" when subtitles are displayed every two lines as in the conventional method. This data will be discussed later.

[0029] The subtitle generation unit 130 starts a timer 300 when the recognition result is saved at the beginning of the stock 250, and generates and outputs subtitle data using the text of the recognized word at predetermined intervals (hereinafter also called the "update frequency") t (for example, 0.2 seconds) based on the timer 300, and resets the timer 300. In the following explanation, we will use the example of displaying a maximum of 15 characters x 2 lines, or a maximum of 30 characters, on a single screen, but the same applies to cases where the maximum number of characters is other than 30. Specifically, the subtitle generation unit 130 starts a timer 300 when, for example, the word "present" is saved at the beginning of an empty stock 250. At the "recognition confirmation time" of the recognition result, when the value of timer 300 reaches an update frequency t (for example, 0.2 seconds) from the recognition confirmation time of the word "present" (time 00:10:27.742), it outputs subtitle data (first row on the left in Figure 3) consisting of words from "present" to the word "today" which has been confirmed and saved in stock 250 during that time, and then resets timer 300. As shown in Figure 2A, there is a difference of approximately 0.6 seconds between the recognition confirmation time of the word "today" and the recognition confirmation time of the next word "daytime". Therefore, until the word "daytime" is added to the stock 250, the subtitle generation unit 130 outputs the subtitle data of the same text from the word "present" to the word "today" stored in the stock 250 each time the value of the timer 300 reaches the update frequency t, and resets the timer 300. In this way, when displaying subtitles sequentially, it is preferable to pre-set the update frequency t (for example, 0.2 seconds) considering that increasing the subtitle update frequency increases the amount of subtitle data to be transmitted (even if no new recognition results are available, it is necessary to generate and transmit subtitle data at each update interval), and that decreasing the subtitle update frequency increases the delay from speech to subtitle display. The subtitle output device 10 can then display subtitles on a television receiver (not shown) without increasing the amount of subtitle data unnecessarily and without increasing the delay from speech to subtitle display.

[0030] Next, when the value of the timer 300 reaches the update frequency t (time 00:10:28.342), the subtitle generation unit 130 outputs subtitle data (second row from the left in Figure 3) in which the word "daytime," which was confirmed and newly saved in the stock 250 during that time, has been added to the text from the word "present" to the word "today," and then resets the timer 300. Next, when the value of the timer 300 reaches the update frequency t (time 00:10:28.542), the subtitle generation unit 130 outputs subtitle data (third row from the left in Figure 3) in which the words "no" and "maximum temperature," which were confirmed and newly saved in the stock 250 during that time, have been added to the text from the word "currently" to the word "daytime," and then resets the timer 300. As shown in Figure 2A, there is a difference of approximately 1 second between the recognition confirmation time of the word "maximum temperature" and the recognition confirmation time of the next word "28". Therefore, until the word "28" is added to the stock 250, the subtitle generation unit 130 outputs the subtitle data of the same text from the word "current" to the word "maximum temperature" stored in the stock 250 each time the value of the timer 300 reaches the update frequency t, and resets the timer 300. Next, when the timer 300 reaches the update frequency t (time 00:10:29.542), the subtitle generation unit 130 outputs subtitle data (fourth row from the left in Figure 3) in which the words "28" and "degrees", which were confirmed and newly saved in stock 250 during that time, have been added to the text from the word "current" to the word "maximum temperature", and then resets the timer 300.

[0031] Next, when the value of the timer 300 reaches the update frequency t (time 00:10:29.742), the subtitle generation unit 130 outputs subtitle data (the 5th row from the left in FIG. 3) obtained by adding the words "gurai" and "made", which were determined and newly saved in the stock 250 during that time, to the text from the word "present" to the word "do", and resets the timer 300. As shown in FIG. 2A, there is a difference of about 0.5 seconds between the recognition determination time of the word "made" and the recognition determination time of the next word "temperature". Therefore, until the character "ki" of the word "temperature" is added to the stock 250, the subtitle generation unit 130 outputs subtitle data of the same text from the word "present" to the word "made" saved in the stock 250 every time the value of the timer 300 reaches the update frequency t, and resets the timer 300. Next, when the recognition of the word "temperature" is determined by the speech recognition unit 120 (time 00:10:30.154), since the maximum number of characters of 30 from the word "present" to the character "ki" of the word "temperature" is saved in the stock 250, regardless of the value of the timer 300, the subtitle generation unit 130 outputs subtitle data of the text from the word "present" to the character "ki" of the word "temperature" (the 6th row from the left in FIG. 3). Then, the subtitle generation unit 130 deletes the recognition result from the word "present" to the character "ki" of the word "temperature" from the stock 250, and resets the timer 300. Note that only the character "on" of the word "temperature" is saved in the stock 250. In this case, as will be described later, the subtitle output device 10 outputs the subtitle data of the 6th row from the left in FIG. 3 until the display time t_d of the word "present" saved at the top of the stock 250, calculated by the display time calculation unit 140, becomes equal to the maximum number of characters that can be displayed on one screen, that is, the total speech time taken for the speech from the word "present" to the word "temperature". That is, as shown in FIG. 2A, since the speaking time of the word "current" is 00:10:25.180 and the speaking time of the next word "is" of the word "temperature" (i.e., the end time of the speaking of the word "temperature") is 00:10:28.720, the total speaking time taken for the speaking from the word "current" to the word "temperature" is 3.54 seconds. On the other hand, the subtitle in the first row on the left side of FIG. 3 is displayed at 00:10:27.742, and since it switched from the subtitle in the sixth row on the left side of FIG. 3 to the subtitle in the first row on the right side of FIG. 3 at 00:10:31.282, the display time t_d is 3.54 seconds, which is equal to the above total speaking time of 3.54 seconds. By doing so, the subtitle output device 10 can display subtitles according to the speaking speed of the speaker, and can also display subtitles that are easy to view for the viewers of the TV program. Also, as shown in FIG. 2A, compared with the conventional subtitles every two lines that are displayed at the time 00:10:30.154 when the recognition of the word "temperature" is confirmed, the subtitle output device 10 can reduce the delay from the speaking to the display of the subtitles. Further, the subtitle output device 10 can balance the data amount and the delay amount of the subtitles by adjusting the update frequency t (for example, 0.2 seconds, etc.).

[0032] Next, when the above display time t_d calculated by the display time calculation unit 140 described later has elapsed (time 00:10:31.282), the subtitle generation unit 130 outputs subtitle data (the first row on the right side of FIG. 3) of the text in which the words from "is" to "during the day" newly saved in the stock 250 and determined during that period are added to the "temperature" of the word "temperature", and resets the timer 300. Next, when the value of the timer 300 becomes the update frequency t (time 00:10:31.482), the subtitle generation unit 130 outputs subtitle data (the second row on the right side of FIG. 3) of the text in which the word "is" newly saved in the stock 250 and determined during that period is added to the text from the "temperature" of the word "temperature" to "during the day", and resets the timer 300. As shown in FIG. 2B, there is a difference of about 0.7 seconds between the recognition determination time of the word "noon" and the recognition determination time of the next word "is". Therefore, until the word "is" is added to the stock 250, the subtitle data of the same text from the "n" in the word "temperature" stored in the stock 250 to the word "noon" is output every time the value of the timer 300 is updated at the update frequency t, and the timer 300 is reset. Next, when the value of the timer 300 reaches the update frequency t (time 00:10:32.082), the subtitle generation unit 130 outputs subtitle data (the third row on the right side of FIG. 3) in which the words "7", "month", and "of" that have been determined and newly stored in the stock 250 are added to the text from the "n" in the word "temperature" to the word "is", and resets the timer 300.

[0033] Next, when the value of the timer 300 reaches the update frequency t (time 00:10:32.282), the subtitle generation unit 130 outputs subtitle data (the fourth row on the right side of FIG. 3) in which the word "early ten days" that has been determined and newly stored in the stock 250 is added to the text from the "n" in the word "temperature" to the word "of", and resets the timer 300. Next, when the value of the timer 300 reaches the update frequency t (time 00:10:32.482), the subtitle generation unit 130 outputs subtitle data (the fifth row on the right side of FIG. 3) in which the word "around" that has been determined and newly stored in the stock 250 is added to the text from the "n" in the word "temperature" to the word "early ten days", and resets the timer 300. As shown in FIG. 2B, there is a difference of about 0.5 seconds between the recognition determination time of the word "around" and the recognition determination time of the next word "of". Therefore, until the word "of" is added to the stock 250, the subtitle data of the same text from the "n" in the word "temperature" stored in the stock 250 to the word "around" is output every time the value of the timer 300 is updated at the update frequency t, and the timer 300 is reset. Next, the subtitle generation unit 130, when the value of timer 300 reaches the update frequency t, has saved the words "no" to "." to stock 250, and has received a silent section (end of speech) as a recognition result from the speech recognition unit 120, outputs subtitle data (sixth row from the right in Figure 3) of the text of all words from "on" to "." in the word "onkōtsu" (temperature) stored in stock 250, regardless of the value of timer 300. Then, the subtitle generation unit 130 deletes all recognition results to empty stock 250 and resets timer 300. In this case, the subtitle output device 10 outputs the sixth line of subtitle data on the right side of Figure 3 until the display time t_d of the word "warm" in the word "temperature" stored at the beginning of the stock 250 calculated by the display time calculation unit 140, as described later, becomes equal to the total utterance time from the word "temperature" to the word ".". In other words, as shown in Figure 2B, the utterance time of the word "temperature" is 00:10:28.580, and the utterance time of the word "." ends at 00:10:32.670. Therefore, the total utterance time from the word "temperature" to the word "." is 4.09 seconds. On the other hand, the first subtitle on the right side of Figure 3 is displayed at 00:10:31.282, and the sixth subtitle on the right side of Figure 3 is displayed at 00:10:32.882. Therefore, if the display time t_d is the total utterance time of 4.09 seconds, the sixth subtitle on the right side of Figure 3 will be displayed until 00:10:35.372 and then disappear. By doing so, the subtitle output device 10 can display subtitles that correspond to the speaker's speaking speed, making them easier for television viewers to read. Also, as shown in Figure 2B, compared to conventional subtitles that appear every two lines at 00:10:32.986 after the word "." is spoken, the subtitle output device 10 can reduce the delay between speaking and displaying the subtitles. Furthermore, the subtitle output device 10 can balance the amount of subtitle data and the delay by adjusting the update frequency t (for example, 0.2 seconds).

[0034] The display time calculation unit 140 calculates, for example, the display time of the subtitle data generated and output by the subtitle generation unit 130 based on the recognition results of the speech recognition unit 120. Specifically, the display time calculation unit 140 calculates, for example, the time at which the total display time of the word stored at the beginning of the stock 250 among the text of the subtitle data generated by the subtitle generation unit 130 (for example, "current") equals the total utterance time for the maximum number of characters that can be displayed on one screen (i.e., from the word "current" to the word "temperature"), based on the utterance time for each word obtained by the speech recognition processing of the speech recognition unit 120 (Figures 2A and 2B), and sets this time as the display time t_d. In other words, as shown in Figure 2A, the utterance time of the word "currently" is 00:10:25.180, and the utterance time of the word "is" following "temperature" (i.e., the end time of the utterance of the word "temperature") is 00:10:28.720. Therefore, the total utterance time from the word "currently" to the word "temperature" is 3.54 seconds. On the other hand, the first subtitle on the left side of Figure 3 is displayed at time 00:10:27.742, and the subtitle switched from the sixth subtitle on the left side of Figure 3 to the first subtitle on the right side of Figure 3 at time 00:10:31.282. Therefore, the display time t_d is 3.54 seconds, which is equal to the total utterance time of 3.54 seconds mentioned above. In Figure 2A, the display time calculation unit 140 calculated a display time t_d equal to the total utterance time, but it is also possible to calculate a display time t_d shorter than the total utterance time depending on the delay amount.

[0035] Furthermore, if the display time calculation unit 140 receives a silent section (end of utterance) as a recognition result from the speech recognition unit 120, it may calculate the display time t_d as the time at which the total display time of the word stored at the beginning of the stock 250 in the text of the subtitle data generated by the subtitle generation unit 130 (for example, "warm" in "temperature") equals the total utterance time of all words stored in the stock 250 (for example, from the word "temperature" to the word "."). In other words, as shown in Figure 2B, the utterance time of the word "temperature" is 00:10:28.580, and the utterance time of the word "." ends at 00:10:32.670. Therefore, the total utterance time from the word "temperature" to the word "." is 4.09 seconds. On the other hand, the first subtitle on the right side of Figure 3 is displayed at 00:10:31.282, and the sixth subtitle on the right side of Figure 3 is displayed at 00:10:32.882. Therefore, if the display time t_d is the total utterance time of 4.09 seconds, the sixth subtitle on the right side of Figure 3 will be displayed until 00:10:35.372 and then disappear. By doing so, the subtitle output device 10 can calculate the display time t_d based on the speaker's speaking speed (speaking time), thereby displaying subtitles that correspond to the speaker's speaking speed, and displaying subtitles that are easy for television viewers to read.

[0036] <Subtitle output processing of subtitle output device 10> Next, referring to Figure 4, we will explain the subtitle output process of the subtitle output device 10. Figure 4 is a flowchart illustrating the subtitle output process of the subtitle output device 10. The flow shown here is executed repeatedly while the television program is running.

[0037] In step S1, the subtitle generation unit 130 determines whether or not it has obtained the recognition result of speech recognition by the speech recognition unit 120 for the audio file acquired by the audio acquisition unit 110. If a recognition result is obtained, the process proceeds to step S2. On the other hand, if no recognition result is obtained, the process proceeds to step S5.

[0038] In step S2, the subtitle generation unit 130 determines whether the recognition result obtained in step S1 is a silent section or not. If the recognition result is a silent section, the process proceeds to step S4. On the other hand, if the recognition result is a speech section and not a silent section, the process proceeds to step S3.

[0039] In step S3, the subtitle generation unit 130 uses the recognition results to perform subtitle output processing for the speech section, generates and outputs subtitle data, and resets the timer 300. Then, the process returns to step S1. The detailed flow of the subtitle output processing for the speech section will be described later.

[0040] In step S4, the subtitle generation unit 130 uses the recognition results to perform subtitle output processing for silent sections, generates and outputs subtitle data, and resets the timer 300. Then, the process returns to step S1. The detailed flow of the subtitle output processing for silent sections will be described later.

[0041] In step S5, the subtitle generation unit 130 determines whether the television broadcast has ended based on pre-set information about the television program (for example, the start and end times of the television program). If the television broadcast has ended, the subtitle output device 10 terminates the subtitle output process. On the other hand, if the television broadcast has not ended, the process proceeds to step S6.

[0042] In step S6, the subtitle generation unit 130 determines whether the value of the timer 300 has reached a preset update frequency t (for example, 0.2 seconds). If the value of the timer 300 has reached the update frequency t, the process proceeds to step S7. On the other hand, if the value of the timer 300 is less than the update frequency t, the process returns to step S1.

[0043] In step S7, the subtitle generation unit 130 generates and outputs subtitle data of the recognized words stored in the stock 250.

[0044] In step S8, the subtitle generation unit 130 resets the timer 300.

[0045] Figure 5 is a flowchart illustrating the detailed processing steps for subtitle output for the speech segment shown in step S3 in Figure 4.

[0046] In step S301, the subtitle generation unit 130 adds the recognized words obtained in step S1 to the stock 250 and saves them.

[0047] In step S302, the subtitle generation unit 130 determines whether the recognition result saved in step S301 is the one saved at the beginning of the stock 250. If the recognition result is the one saved at the beginning of the stock 250, the process proceeds to step S303. On the other hand, if the recognition result is not the one saved at the beginning of the stock 250, the process proceeds to step S304.

[0048] In step S303, the subtitle generation unit 130 starts the timer 300.

[0049] In step S304, the subtitle generation unit 130 determines whether the number of words stored in the stock 250 exceeds the maximum number of characters (15 characters x 2 lines) on one screen, i.e., 30 characters. If the number of words exceeds the maximum number of characters, the process proceeds to step S305. On the other hand, if the number of words is less than or equal to the maximum number of characters, the process proceeds to step S308.

[0050] In step S305, the display time calculation unit 140 calculates the display time t_d of the subtitle data generated by the subtitle generation unit 130 based on the recognition result of the speech recognition unit 120.

[0051] In step S306, the subtitle generation unit 130 generates and outputs subtitle data using 15 characters x 2 lines, i.e., a maximum of 30 characters, from the recognized words stored in the stock 250.

[0052] In step S307, the subtitle generation unit 130 deletes the recognition result used for the subtitle data generated in step S306 from the stock 250 and resets the timer 300.

[0053] In step S308, the subtitle generation unit 130 determines whether the value of the timer 300 has reached the update frequency t (for example, 0.2 seconds). If the value of the timer 300 has reached the update frequency t, the process proceeds to step S309. On the other hand, if the value of the timer 300 is less than the update frequency t, the process terminates the subtitle output process for the speech section and returns to the process in step S1.

[0054] In step S309, the subtitle generation unit 130 generates and outputs subtitle data of the recognized words stored in the stock 250.

[0055] In step S310, the subtitle generation unit 130 resets the timer 300. Then, the subtitle output device 10 returns to the process of step S1.

[0056] Figure 6 is a flowchart illustrating the detailed processing steps for outputting subtitles during the silent section shown in step S4 in Figure 4.

[0057] In step S401, the subtitle generation unit 130 determines whether or not the recognition result is stored in the stock 250. If the recognition result is stored in the stock 250, the process proceeds to step S402. On the other hand, if the recognition result is not stored in the stock 250, the subtitle output device 10 terminates the subtitle output process for the silent section and returns to the process in step S1.

[0058] In step S402, the display time calculation unit 140 calculates the display time t_d of the subtitle data generated by the subtitle generation unit 130 based on the recognition result of the speech recognition unit 120.

[0059] In step S403, the subtitle generation unit 130 generates and outputs subtitle data of the text of all recognized words stored in the stock 250.

[0060] In step S404, the subtitle generation unit 130 deletes all recognition results stored in the stock 250 and resets the timer 300. Then, the subtitle output device 10 returns to the process of step S1.

[0061] As described above, the subtitle output device 10 according to one embodiment can reduce the delay between speech and subtitle display by displaying subtitles when speech recognition of a portion of the subtitles to be displayed on the screen has been determined, based on the update frequency t (for example, 0.2 seconds), the number of words that can be displayed on one screen, or the recognition results of silent sections, and updating the subtitles each time a new recognition result is determined. Furthermore, the subtitle output device 10 can balance the amount of subtitle data and the delay between speech and subtitles by adjusting the value of the update frequency t.

[0062] Although one embodiment has been described above, the subtitle output device 10 is not limited to the embodiment described above, and may include modifications, improvements, etc., to the extent that the objective can be achieved.

[0063] <Example 1> In one embodiment, the display time calculation unit 140 calculates the display time t_d such that the total display time of the first word stored in the stock 250 is equal to the total utterance time of all words displayed as subtitles, but is not limited to this. For example, the display time calculation unit 140 may calculate a lower limit for the display time of the last word displayed on a single screen. By doing so, the subtitle output device 10 can also prevent the subtitle display time from becoming too short.

[0064] <Modification 2> Furthermore, in the above-described embodiment, the subtitle generation unit 130 generated subtitle data and output it to the video generation device 30 based on the update frequency t (e.g., 0.2 seconds), the number of words that can be displayed on one screen, or the recognition result of silent sections, but is not limited to this. For example, the subtitle generation unit 130 may, when outputting subtitle data, wait until the currently displayed subtitles are finished, and then output new subtitle data simultaneously with the end of the display. If new recognition results are determined during this waiting period, the subtitle generation unit 130 may add the words of the determined recognition results to the memory unit 200 and reflect them in the subtitle data to be displayed next.

[0065] <Variation 3> Furthermore, while the subtitle output device 10 generated subtitles for a television program in the above-described embodiment, it is not limited to this. For example, the subtitle output device 10 can also be applied to movies, online streaming videos, and the like.

[0066] Each function included in the subtitle output device 10 in one embodiment can be implemented by hardware, software, or a combination thereof. Here, implementation by software means that it is implemented by a computer loading and executing a program.

[0067] Programs can be stored and supplied to a computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Programs may also be supplied to a computer using various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0068] Furthermore, the step of writing the program to be recorded on the recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually. [Explanation of Symbols]

[0069] 1. Subtitle Output System 10 Subtitle output device 100 Control Unit 110 Voice acquisition unit 120 Voice Recognition Unit 130 Subtitle generation section 140 Display time calculation section 200 Storage section 250 stock 300 timer 20 Voice input device 30 Video generation device

Claims

1. A speech recognition unit performs speech recognition processing on the input audio, Each time the recognition result recognized by the speech recognition unit is finalized, a stock is set up to store the recognition result. Timer and A subtitle generation unit that starts the timer when the recognition result is saved at the beginning of the stock, and generates and outputs subtitle data based on the recognition result stored within a predetermined time interval based on the timer, A subtitle display device equipped with a subtitle display.

2. The subtitle display device according to claim 1, further comprising a display time calculation unit that calculates the display time of the subtitle data generated and output by the subtitle generation unit based on the recognition result of the speech recognition unit.

3. The subtitle generation unit, when the speech recognition unit recognizes a silent section, generates and outputs subtitle data based on the recognition result stored in the stock regardless of the timer value, deletes all stored recognition results from the stock, and resets the timer, as described in claim 1 or claim 2.

4. The subtitle generation unit, when the amount of data of the recognition results stored in the stock exceeds the amount of data that can be displayed on one screen, generates and outputs subtitle data of a number of characters that can be displayed on one screen from the recognition results stored in the stock, regardless of the value of the timer, deletes the recognition results used in the output subtitle data from the stock, and resets the timer, as described in claim 1 or claim 2.

5. As a subtitle display device for computers, A speech recognition function that performs speech recognition processing on input audio. A stock function that saves the recognition result each time the speech recognition result is confirmed. The first function is a subtitle generation function that starts a timer included in the subtitle display device when the recognition result is saved, and generates and outputs subtitle data based on the recognition result stored within a predetermined time interval based on the timer. A subtitle display program to achieve this.

Images