Subtitle generation device, method, and program

Abstract

To provide a subtitle generator, method, and program that reduce the burden of correcting subtitles to make them easier to read when generating subtitles from speech recognition results in real time.SOLUTION: A subtitle generator includes: an acquisition unit that sequentially acquires text of speech recognition results; a history unit that stores the text as history data; a generation unit that estimates division and joint positions of the text from one or more stored history data and generates subtitle text from one or more history data based on the division and joint positions; a history update unit that updates history data based on division and joint positions; and a user interface unit that presents subtitle text.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a subtitle generation device, method, and program. [Background technology]

[0002] In the production of subtitles for television programs and other programs, there is a technology that uses speech recognition to automatically generate subtitles. While the accuracy of speech recognition has rapidly improved in recent years thanks to advances in deep learning technology, there is still the possibility of speech recognition errors due to background noise, slip-ups, and speech containing technical terms. For this reason, in actual subtitle production, not all subtitles are produced using speech recognition; in most cases, the subtitles resulting from speech recognition are manually reviewed and errors corrected before the final subtitles are created.

[0003] Furthermore, when generating subtitles from speech recognition results in real time, such as for news programs, obtaining speech recognition results early is desirable to ensure time for manual correction. One effective method for achieving this is a technique called early confirmation. Early confirmation is a technique that terminates search and outputs recognition results at the point when the number of candidates in the search space becomes limited in pattern recognition, including speech recognition. This technique is useful for applications that use speech recognition in real time. However, when early confirmation is used, the search is terminated before the end of the utterance, which can result in the speech recognition results being cut short, or conversely, the search space cannot be narrowed and the speech recognition results can be long. Therefore, when using early confirmation technology for subtitle generation, the burden of correcting the results by combining or dividing them to make the subtitles easier for humans to read is significant.

[0004] As described above, when conventional caption generation devices generate captions from speech recognition results in real time, the burden of correcting the captions to make them easier to read is heavy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6562698 [Non-patent literature]

[0006] [Non-Patent Document 1] Tomoyasu Komori, "Latest Trends in Automatic Subtitling for Live Broadcast Programs," NHK Science & Technology Research Institute R&D, NHK Broadcasting Technology Research Institute, Summer 2020, No. 182, [Retrieved September 7, 2021], Internet<URL:https: / / www.nhk.or.jp / strl / publica / rd / 182 / 3.html> [Non-patent document 2] Tomoaki Iizumi and two others, "Line Break Insertion into Lecture Text Using RNN for Readable Subtitle Generation," Proceedings of the 82nd National Conference of the Information Processing Society of Japan, March 6, 2020 Summary of the Invention [Problem to be solved by the invention]

[0007] The problem that the present invention aims to solve is to provide a subtitle generation device, method, and program that can reduce the burden of corrections to make subtitles easier to read when generating subtitles from speech recognition results in real time. [Means for solving the problem]

[0008] A subtitle production device according to an embodiment includes an acquisition unit, a history unit, a generation unit, a history update unit, and a presentation unit. The acquisition unit sequentially acquires text resulting from speech recognition. The history unit stores the text as history data. The generation unit estimates division and combination positions of the text from the one or more pieces of history data stored, and generates subtitle text from the one or more pieces of history data based on the division and combination positions. The history update unit updates the history data based on the division and combination positions. The presentation unit presents the subtitle text. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a block diagram illustrating an example of a subtitle production device according to a first embodiment. [Figure 2] FIG. 3 is a schematic diagram for explaining history data according to the first embodiment. [Figure 3] 5 is a flowchart illustrating an example of an operation according to the first embodiment. [Figure 4] FIG. 3 is a schematic diagram for explaining an example of an operation in the first embodiment. [Figure 5] FIG. 10 is a block diagram illustrating an example of a subtitle production device according to a second embodiment. [Figure 6] 10 is a flowchart illustrating an example of an operation according to the second embodiment. [Figure 7] FIG. 10 is a schematic diagram for explaining an example of an operation in the second embodiment. [Figure 8] FIG. 10 is a block diagram illustrating an example of a subtitle production device according to a third embodiment. [Figure 9] 10 is a flowchart illustrating an example of an operation according to the third embodiment. [Figure 10] FIG. 11 is a schematic diagram for explaining an example of an operation according to the third embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the hardware configuration of a caption production device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Each embodiment will be described below with reference to the drawings. The subtitle generation device according to each embodiment is used in a system for producing subtitles for television news programs and the like, where audio is recognized in real time and then manually checked and corrected. Note that "real time" in each embodiment does not refer to instantaneous or simultaneous time, but includes delays due to data processing and communication. Furthermore, the term "subtitle generation device" may be appropriately replaced with any other name, such as an automatic subtitle generation device.

[0011] First Embodiment Fig. 1 is a block diagram showing a subtitle production device according to a first embodiment. In Fig. 1, a speech recognition unit 10 is connected to a subtitle production device 20. The subtitle production device 20 includes a history unit 21, an acquisition unit 22, a history update unit 23, a subtitle processing unit 24, and a user interface unit 25. The subtitle processing unit 24 includes a generation unit 24a.

[0012] Here, the speech recognition unit 10 continuously acquires speech data, performs speech recognition on the acquired speech data, and generates text as a result of the speech recognition.

[0013] The history unit 21 is a memory that can be read / written by the history update unit 23 and the generation unit 24a. The history unit 21 stores, for example, history data related to the text of the acquired speech recognition result and data related to the generation of subtitle text based on the history data. As shown in FIG. 2, the history data 21a is information that associates the text of the speech recognition result sequentially acquired by the acquisition unit 22 with the number of output characters and the position of divided characters, which are information related to the subtitle text generated by the generation unit 24a. Note that the history data 21a may further include, for each text of the speech recognition result, an identification ID, an acquisition time (acquisition date and time), a character string length (number of characters), etc., in addition to the number of output characters and the position of divided characters.

[0014] The identification ID is information for identifying each of the sequentially acquired texts.

[0015] The acquisition time is the time when each text was acquired and includes the acquisition date (not shown). That is, the acquisition time is an example of the acquisition date and time.

[0016] The string length is the length of the string of text, in other words, the number of characters in the text.

[0017] The number of characters that have been output is the number of characters of the text that have been output from the generation unit 24a to the user interface unit 25.

[0018] The dividing character position is the position of a character that divides (breaks a line in) the text when the text is combined.

[0019] Data related to generating subtitle text can include the current position CSI of the display candidate, the display candidate sequence, the string length of the concatenated text, the maximum number of characters, the word identification WID, the feature vector, the update information, the segmentation estimation score, and the word segmentation position SWI, as described below.

[0020] The acquisition unit 22 sequentially acquires text of the speech recognition result from the speech recognition unit 10. Note that the subtitle production device 20 may include, instead of the speech recognition unit 10 and the acquisition unit 22, a speech recognition unit that continuously acquires speech data, performs speech recognition on the acquired speech data, and sequentially sends text of the speech recognition result to the history update unit 23.

[0021] The history update unit 23 stores the text of the speech recognition result acquired by the acquisition unit 22 as history data 21a in the history unit 21. Furthermore, the history update unit 23 updates the history data 21a in the history unit 21 based on information about the subtitle text generated by the generation unit 24a.

[0022] The generation unit 24a estimates text division and joining positions from one or more history data 21a stored in the history unit 21, and generates subtitle text from the history data 21a based on the estimated division and joining positions. Here, the generation unit 24a may estimate joining positions so that text from the history data is joined until the number of characters in the concatenated text becomes equal to or exceeds a threshold number of characters (maximum number of characters). The generation unit 24a may also perform either or both of morphological analysis and dependency analysis on the concatenated text, and generate a feature vector for each word based on the results of the analysis. In this case, the generation unit 24a calculates a division estimation score for each word using the feature vector, and estimates the division position from the division estimation score.

[0023] The user interface unit 25 presents the subtitle text generated by the generation unit 24a to the user. The user interface unit 25 may be, for example, a display that displays the subtitle text, or a touch panel that serves both as the display and an input function. The user interface unit 25 may also include a keyboard, a mouse, or the like as an input function in addition to the display. The input function can be used as appropriate to correct division positions, joining positions, voice recognition errors, etc. in the subtitle text. The user interface unit 25 is an example of a presentation unit.

[0024] Next, the operation of the caption production device configured as above will be described with reference to the flowchart of FIG. 3 and the schematic diagram of FIG.

[0025] (Step ST1) The acquisition unit 22 sequentially acquires text of the speech recognition result. Specifically, the speech recognition unit 10 performs speech recognition on continuously received speech data, and generates text of the speech recognition result as shown in FIG. 4. The acquisition unit 22 sequentially acquires text of the speech recognition result from the speech recognition unit 10. Note that, as the text to be acquired, text of the speech recognition result using the above-mentioned early determination technique is preferable from the viewpoint of having less time lag. However, the text to be acquired is not limited to this, and the text to be acquired may also be text of the speech recognition result when a general speech segment estimation technique is used.

[0026] (Step ST2) The history update unit 23 stores the acquired text of the speech recognition result as history data 21a in the history unit 21. In the example shown in FIG. 2, in response to the utterance "Reservations for the large-scale vaccination center for the COVID-19 vaccine started today. The Tokyo venue is the Joint Government Building," the speech recognition result texts acquired sequentially, such as "COVID-19 vaccine," "large-scale vaccination," and "center," are stored as history data 21a. The history data 21a stores, along with the speech recognition result text, at least, an identification ID for identifying the history data 21a, the number of characters already output, and the position of the divided characters. When the history update unit 23 acquires the speech recognition text, it creates new data with a new identification ID, a number of characters already output of 0, and no divided character position, along with the speech recognition result text, and stores the data as history data 21a in the history unit 21. The above steps ST1 to ST2 are repeatedly executed in parallel with the processing from the next step ST3 onward.

[0027] (Step ST3) The generating unit 24a extracts, from the history data 21a in the history unit 21, one or more pieces of history data 21a that are not presented as subtitle text.

[0028] For example, the array of history data 21a in the history unit 21 is referred to as a history data array. Specifically, the history data array has seven pieces of history data 21a, each containing text identified by an ID of 0 to 6. Each piece of history data 21a is, for example, {ID=0 and "COVID-19 vaccine"}, {ID=1 and "large-scale vaccination"}, {ID=2 and "of the center"}, {ID=3 and "reservation today"}, {ID=4 and "started from"}, {ID=5 and "Tokyo venue"}, and {ID=6 and "Tokyo joint government building"}. It is also assumed that the maximum number of characters in the subtitle text to be generated is preset to 30 characters.

[0029] First, the generation unit 24a initializes the current position CSI of the display candidate in the history data array, a display candidate array that holds one or more history data 21a in the history data array, and the character string length of the concatenated text that concatenates the text of the history data 21a in the display candidate array.

[0030] Furthermore, if the size of the history data array is larger than the current position CSI of the display candidate, that is, if there is history data 21a in the history data array that is not displayed as subtitle text, the generating unit 24a repeats the following process.

[0031] First, when the generation unit 24a acquires the CSI-th piece of history data 21a from the history data array, it increments the current position CSI of the display candidate by one and adds the history data 21a to the display candidate array.

[0032] Next, the generation unit 24a adds a value obtained by subtracting the number of characters already output in the history data 21a from the character string length of the text of the speech recognition result in the history data 21a to the character string length of the concatenated text. Here, if the character string length of the concatenated text is greater than the maximum number of characters of the subtitle text to be generated, the generation unit 24a proceeds to step ST4 to make the concatenated text easier to read.

[0033] For example, when the current position CSI of the display candidate is 0, the 0th history data 21a (identification ID = 0) "COVID-19 vaccine" is acquired from the history data array, and the acquired history data 21a is added to the display candidate array. At this time, the character string length of the history data 21a with identification ID = 0 is 13, and the number of output characters is 0, so the character string length of the concatenated text is 13. Thereafter, the current position CSI of the display candidate is incremented by 1, and the CSIth history data 21a is acquired and added to the display candidate array in the same manner. As a result, when the character string length of the concatenated text exceeds the maximum character count (30 characters), the history data 21a with identification IDs = 0 to 4 is added to the display candidate array. Note that the text of each history data 21a with identification IDs = 0 to 4 is "COVID-19 vaccine," "mass vaccination," "center," "appointment made today," and "started from." Therefore, in this example, the character string length of the concatenated text is 39 (= 13 + 6 + 5 + 5 + 10).

[0034] (Step ST4) The generation unit 24a estimates joining positions and division positions from the extracted text of the history data 21a. For example, the generation unit 24a estimates joining positions so that the text of the history data 21a is joined until the number of characters in the concatenated text becomes equal to or exceeds the maximum number of characters (threshold number of characters). Also, for example, the generation unit 24a estimates division positions from the display candidate array and proceeds to step ST5. Specifically, for example, the division positions are estimated from the context before and after.

[0035] First, the generating unit 24a initializes the concatenated text obtained by concatenating the texts in the display candidate array and the update information of the history data 21a.

[0036] Next, the generation unit 24a stores in the update information the association between the identification ID of each piece of history data 21a in the display candidate array and the text start position, and concatenates the text until the number of characters reaches or exceeds the maximum number to create concatenated text.

[0037] For example, suppose the text in the history data 21a with IDs 0 to 4 is "COVID-19 vaccine," "mass vaccination," "center," "reservations are open today," and "started today." In this case, the update information stores {ID=0 and text start position "0"}, {ID=1 and text start position "13"}, {ID=2 and text start position "19"}, {ID=3 and text start position "24"}, and {ID=4 and text start position "29"}. Note that the text start position corresponds to the length of the character string preceding each piece of text in the concatenated text. In addition, the concatenated text "Reservations for the COVID-19 vaccine mass vaccination center started today" is created. As described above, the character string length of this concatenated text is 39, and the speech recognition result text is concatenated until it reaches the maximum length of 30 characters.

[0038] Next, the generation unit 24a performs either or both of morphological analysis and dependency analysis on the concatenated text, and generates a feature vector for each word based on the results of the analysis. The generation unit 24a also calculates a segmentation estimation score for each word using the feature vector, and estimates the segmentation position from the segmentation estimation score.

[0039] Specifically, for example, the generating unit 24a performs a morphological analysis on the concatenated text, and generates a word sequence in which each word is associated with a word identification WID from the results of the morphological analysis. For example, the following word sequences are generated: {word identification WID=0 and "new coronavirus"}, {word identification WID=1 and "vaccine"}, {word identification WID=2 and "of"}, {word identification WID=3 and "large-scale"}, {word identification WID=4 and "vaccination"}, {word identification WID=5 and "center"}, {word identification WID=6 and "of"}, {word identification WID=7 and "reservation"}, {word identification WID=8 and "ga"}, {word identification WID=9 and "today"}, {word identification WID=10 and "kara"}, {word identification WID=11 and "start"}, {word identification WID=12 and "shi"}, {word identification WID=13 and "mash"}, {word identification WID=14 and "ta"}.

[0040] Next, the generating unit 24a associates the identification ID of each piece of history data 21a with the word identification WID for the word sequence using the text start position stored in the update information, and stores the result in the update information.

[0041] For example, in the case of historical data 21a with identification ID = 0 to 4 ("COVID-19 vaccine", "large-scale vaccination", "Center", "Reservation today", "Started from"), the following is saved in the update information: {Identification ID = 0 and "New coronavirus (WID = 0)" and "Vaccine (WID = 1)"}, {Identification ID = 1 and "of (WID = 2)", "large-scale (WID = 3)", "vaccination (WID = 4)"}, {Identification ID = 2 and "reservation (WID = 7)", "ga (WID = 8)", "today (WID = 9)"}, {Identification ID = 3 and "center (WID = 5)", "of (WID = 6)"}, {Identification ID = 4 and "from (WID = 10)", "start (WID = 11)", "shi (WID = 12)", "mashi (WID = 13)", "ta (WID = 14)"}.

[0042] Next, the generation unit 24a generates a feature vector for each word in the word sequence. The features used in the feature vector include, for example, a word index from the beginning of each word, character length, part of speech, and dependency analysis results, as appropriate.

[0043] Next, the generation unit 24a calculates a division estimation score for determining whether to perform division before each word using a trained model based on the generated feature vectors. Here, for example, a deep learning model can be used as the trained model as appropriate. For example, an N-gram deep neural network (DNN) model may be used as the deep learning model. In this case, the generation unit 24a may combine the feature vectors of the N words before and after each word into a single vector, input the vector into a DNN model, and calculate a division estimation score for each word using the DNN model.

[0044] Next, the generation unit 24a determines the word division position SWI, which is the position of the word to be divided, from the division estimation score. For example, the word division position SWI is set to be before the word with the largest division estimation score. Alternatively, the word division position SWI is set to be before several words with scores equal to or higher than a threshold.

[0045] Specifically, for example, assume that the word division position SWI=[3,9]. In this case, the word division position SWI indicates that the division occurs before the word "large-scale" with word identification WID=3 and before the word "today" with word identification WID=9. To add, among the word division positions SWI=[3,9], SWI=[3] indicates the space between the word "of" with word identification WID=2 and the word "large-scale vaccination center" with word identification WID=3. Similarly, among SWI=[3,9], SWI=[9] indicates the space between the word "ga" with word identification WID=8 and the word "today" with word identification WID=9.

[0046] Next, once the word sequence and word division positions SWI are obtained, the process proceeds to step ST5 where subtitle text is generated.

[0047] (Step ST5) The generation unit 24a generates subtitle text based on the extracted history data 21a and the estimated joining and division positions. The generation unit 24a also creates update information required to update each piece of history data 21a. For example, the generation unit 24a extracts the word index of each word and update information in which each word appears from the word start position in the update information history data 21a, and creates update information by adding the number of output characters to the corresponding update information.

[0048] For example, let's assume the aforementioned word division position SWI=[3,9]. From the display candidate array including history data 21a with identification IDs = 0 to 4, the generation unit 24a creates update information that holds the subtitle text "Reservation made at the ¥n large-scale vaccination center for the COVID-19 vaccine" and the identification ID and number of output characters for each piece of history data 21a. Note that "¥n" represents a line break at the division position. The update information includes, for example, {identification ID=0 and number of output characters: 13}, {identification ID=1 and number of output characters: 6}, {identification ID=2 and number of output characters: 4}, {identification ID=3 and number of output characters: 3}, and {identification ID=4 and number of output characters: 0}.

[0049] (Step ST6) The user interface unit 25 presents the generated subtitle text to the user. For example, the user interface unit 25 sequentially displays up to two lines of subtitle text as shown in Fig. 4. The generation unit 24a also initializes a display candidate array and the character string length of the concatenated text.

[0050] (Step ST7) The history update unit 23 updates the history data 21a stored in the history unit 21 based on the generated subtitle text. For example, the history update unit 23 updates the history data array based on the update information of the history data 21a. Furthermore, the history update unit 23 updates the current position CSI of the display candidate to a position in the history data 21a where there is an undisplayed item based on the update information.

[0051] For example, in the case of the update information described in step ST5, the current position CSI of the display candidate is updated because the text string length "5" does not match the number of characters already output "3" in the history data 21a of identification ID=3.

[0052] (Step ST9) The generation unit 24a determines whether there is any unpresented history data 21a, and if there is any unpresented history data 21a, repeatedly executes the processes of steps ST3 to ST9. On the other hand, if the determination result shows that there is no unpresented history data 21a, the process ends.

[0053] As described above, according to the first embodiment, the acquisition unit 22 sequentially acquires text of the speech recognition result. The history unit 21 saves the text as history data 21a. The generation unit 24a estimates division and joining positions of the text from the one or more pieces of saved history data 21a. The generation unit 24a also generates subtitle text from the one or more pieces of history data 21a based on the division and joining positions. The history update unit 23 updates the history data 21a based on the division and joining positions. The user interface unit 25, which serves as a presentation unit, presents the subtitle text. Therefore, when generating subtitles from speech recognition results in real time, the burden of corrections to make the subtitles easier to read can be reduced.

[0054] In addition, real-time subtitling using speech recognition technology can automatically generate subtitle text that is easy to read, reducing the revision costs for subtitlers. Furthermore, if speech recognition results using early confirmation technology are used, more time can be spent on revisions.

[0055] Comparative examples (1) to (3) for the first embodiment will be described below.

[0056] (1) The first comparative example is a technology that estimates line break positions for transcribed text to improve the readability of subtitles. To make subtitles of lectures easier to read, the technology of the first comparative example uses a recurrent neural network (RNN), a deep learning technology, to estimate whether to insert a line break before each sentence of the transcribed lecture text. However, because the first comparative example estimates line break positions from the entire transcribed lecture text, it is not suitable for generating subtitles in real time. Specifically, the first comparative example sequentially acquires and displays recognition results, making it a technology that is not suitable for manual correction, making it difficult to apply to real-time subtitle production for news programs, etc.

[0057] (2) The second comparative example is a technology for producing subtitles that adjusts the output playback of audio data based on the number of characters in the text of the speech recognition results. This second comparative example does not affect the readability of the produced subtitles. Therefore, in the second comparative example, subtitles must be manually edited to take readability into consideration, which results in high manual editing costs.

[0058] (3) The third comparative example is a technology for correcting speech recognition results by specifying the correction points on a touch panel, etc., and re-speaking. However, since the third comparative example does not have a configuration for sequential and continuous correction, it cannot be applied to real-time subtitling.

[0059] As described above, none of the comparative examples (1) to (3) are suitable for real-time generation of easy-to-read subtitles, and therefore the effects of the first embodiment described above cannot be obtained.

[0060] Furthermore, according to the first embodiment, the generation unit 24a estimates the joining position so that text from the history data is joined until the number of characters in the concatenated text reaches or exceeds the threshold number of characters (maximum number of characters). Therefore, in addition to the effects described above, since the division position is estimated from concatenated text with a number of characters slightly exceeding the threshold number of characters, it is expected that subtitle text with a number of characters close to the threshold number of characters will be generated. In other words, it is possible to generate as much subtitle text as possible within the displayable range, thereby reducing the delay time in displaying subtitles.

[0061] Furthermore, according to the first embodiment, the generation unit 24a performs either or both of morphological analysis and dependency analysis on the concatenated text obtained by combining texts, and generates a feature vector for each word based on the results of the analysis. The generation unit 24a also calculates a division estimation score for each word using the feature vector, and estimates the division position from the division estimation score. Therefore, in addition to the effects described above, the division position can be estimated from the concatenated text without manual intervention, further reducing the burden of editing subtitles to make them easier to read.

[0062] (Modification of the first embodiment) The first embodiment may be implemented as the following modified example, which can be similarly applied to the following embodiments.

[0063] That is, although feature vectors are used in the first embodiment, this is not limiting. For example, word vectors using a word embedding technique may be used instead of feature vectors. In this case, the generation unit 24a performs either or both of morphological analysis and dependency analysis on the concatenated text obtained by combining texts, generates an embedding vector for each word based on the execution results, calculates a division estimation score for each word using the embedding vector, and estimates the division position from the division estimation score. Even in this way, the same effect as in the first embodiment can be obtained.

[0064] Next, in the first embodiment, an N-gram DNN model is used, but this is not limiting. For example, a recurrent neural network (RNN) model may be used instead of the N-gram DNN model. In this case, an RNN model that receives the feature vector of each word as input and calculates the segmentation estimation score of each word may be used. The RNN model may be unidirectional, taking into account the context on one side of the word of interest, or bidirectional, taking into account the context on both sides. Specifically, for example, when calculating the segmentation estimation score of each word, the generation unit 24a may calculate the segmentation estimation score using an LSTM model that takes into account the context before each word, the context after each word, or the contexts before and after each word. Note that LSTM is an abbreviation for "Long Short-Term Memory."

[0065] Furthermore, in the first embodiment, the process of generating subtitle text is performed on a word-by-word basis, but this is not limiting. For example, instead of processing on a word-by-word basis, a phrase formed by grouping words may be used as a processing unit, and a feature vector grouped on each phrase may be used. In this case, the generation unit 24a performs either or both of morphological analysis and dependency analysis on the concatenated text formed by combining texts, groups the results of the analysis on a phrase-by-phrase basis, generates a feature vector for each phrase, calculates a segmentation estimation score for each phrase using the feature vector, and estimates the segmentation position from the segmentation estimation score. This method can also achieve the same effects as the first embodiment.

[0066] In the first embodiment, unnecessary words, such as parts of speech such as unnecessary fillers and symbols, and specific character strings, are not deleted when generating subtitle text. However, this is not limiting. That is, the configuration may be such that the unnecessary words are deleted when generating subtitle text. In this case, the generation unit 24a performs a process of deleting the unnecessary words after morphological analysis, includes the positions of the deleted words in the text in the update data, and reflects this in each history data when updating the history.

[0067] In the first embodiment, in the process of generating subtitle text, a division estimation score is calculated and a division position is estimated from the division estimation score, but this is not limiting. For example, in the process of generating subtitle text, the division position may be determined in advance, and the subtitle text may be divided at the predetermined fixed division position.

[0068] In the first embodiment, the generation unit 24a estimates the joining positions so as to join texts in the history data 21a until the number of characters in the concatenated text becomes equal to or greater than a threshold number of characters (maximum number of characters), but this is not limiting. For example, the generation unit 24a may estimate the division positions for the concatenated text according to a fixed number of characters that is different from the threshold number of characters. The fixed number of characters is, for example, an arbitrary number of characters that is smaller than the maximum number of characters.

[0069] In the first embodiment, the division positions are estimated regardless of the number of pieces of history data not displayed in the history data array. However, this is not limiting. That is, the model or process for estimating the division positions may be changed depending on the number of pieces of history data not displayed in the history data array. Supplementally, for example, when the number of pieces of history data 21a is increasing, the generation unit 24a estimates the division positions based on a fixed number of characters, which requires less calculation, because it is better to present the pieces of history data 21a to the user sooner. That is, when the number of pieces of history data 21a stored in the history unit 21 that include text not presented as subtitle text is equal to or greater than a threshold, the generation unit 24a estimates the division positions based on the fixed number of characters. On the other hand, when the number of pieces of history data 21a falls below a certain level, the generation unit 24a calculates a division estimation score using a division estimation model that requires a large amount of calculation but has high accuracy. That is, when the number of pieces of history data 21a including text not presented as subtitle text is less than a threshold, the generation unit 24a estimates the division positions by calculating a division estimation score.

[0070] Furthermore, in the first embodiment, when estimating the division position, the division position may be estimated within a range longer than the range displayed on one screen, including both the front and back. Specifically, for example, in determining whether to add history data 21a to the display candidate array, history data 21a is included in the display candidate array until the number of characters in the concatenated text is equal to or greater than the maximum number of characters displayed on one screen plus a few characters (+α characters), and the division position is estimated in the same manner as described above. In this case, when generating subtitle text, the division position may be estimated so that the portion of the first character string that has been displayed in the past and the portion of the last character string that exceeds the maximum number of characters are not included in the subtitle text.

[0071] In the first embodiment, the determination of whether to add history data 21a to the display candidate array is based on the maximum number of characters that can be displayed on one screen. However, this is not limiting. For example, instead of the method of determining based on the maximum number of characters, a method may be used in which the acquisition date and time of the text of the speech recognition result is associated with the text and stored in the history unit 21, and history data that has not been displayed even after a certain time has elapsed since the acquisition date and time is added to the display candidate array. In this case, the history data 21a includes the text of the speech recognition result and the acquisition date and time of the text. The generation unit 24a estimates the combining position so as to combine the text of the history data 21a that has not been displayed even after a certain time has elapsed since the acquisition date and time. The certain time is, for example, 10 seconds. However, the certain time is not limited to this, and any time within a range of, for example, 1 to 10 seconds can be used as the certain time.

[0072] <Second embodiment> Compared to the first embodiment, the second embodiment adds a configuration for correcting the division and joining positions of the subtitle text in accordance with a user action (operation) on the subtitle text portion displayed on the user interface unit 25. Note that the word "action" may be read as "operation."

[0073] 5 is a block diagram showing a subtitle production device according to a second embodiment, in which the same components as those in FIG. 1 are given the same reference numerals and detailed descriptions thereof are omitted, and the following mainly describes the different parts. Similarly, duplicated descriptions will be omitted for each of the following embodiments.

[0074] Compared to the configuration shown in Figure 1, the subtitle production device 20 shown in Figure 5 further includes a regeneration unit 24b in the subtitle processing unit 24, and further includes a correction request receiving unit 26 between the regeneration unit 24b and the user interface unit 25.

[0075] Accordingly, in addition to the above-mentioned configuration, the user interface unit 25 detects actions taken by the user on parts of the presented subtitle text and generates action data. This user interface unit 25 is an example of a detection unit.

[0076] The correction request receiving unit 26 determines the range of subtitle text to be corrected and the type of request, either division or joining, for the subtitle text, based on the action data. Here, the correction request receiving unit 26 makes this determination, for example, when it receives action data as a correction request. This correction request receiving unit 26 is an example of a determination unit.

[0077] The regenerator 24b re-estimates the division or joining positions of the subtitle text based on the correction target range, the desired type, and the history data 21a, and re-generates the subtitle text from the re-estimated results.

[0078] The operation of the caption production device configured as above will be described with reference to the flowchart of Fig. 6 and the schematic diagram of Fig. 7. The operation of this embodiment is executed as step ST8 between the above-mentioned step ST7 and step ST9. Step ST8 is executed as the following steps ST8-1 to ST8-6.

[0079] (Step ST8-1) When the user performs an action on a part of the subtitle text displayed in step ST6, the user interface unit 25 detects the action and generates action data. Specifically, it is assumed that an action such as cursor movement, clicking, dragging, flicking, or swiping is performed on the displayed subtitle text. In this case, the user interface unit 25 generates information such as the start position and end position on the screen where the action was performed as action data.

[0080] (Step ST8-2) The user interface unit 25 notifies the action data to the correction request receiving unit 26. The correction request receiving unit 26 receives the action data.

[0081] (Step ST8-3) Next, the correction request receiving unit 26 determines the range of subtitle text to be corrected and whether the requested type is division or joining of subtitle text, based on the action data.

[0082] (Step ST8-4) The regenerator 24b re-estimates the division or joining positions of the subtitle text based on the correction range of the subtitle text, the request type, and the history data 21a, and re-generates the subtitle text from the re-estimated results.

[0083] (Step ST8-5) The user interface unit 25 presents the regenerated subtitle text to the user.

[0084] (Step ST8-6) Based on the regenerated subtitle text, the history update unit 23 updates the history data 21a stored in the history unit 21. As a result, the subtitle production device 20 ends step ST8, which is made up of steps ST8-1 to ST8-6, and proceeds to step ST9 described above.

[0085] Next, the above-described operations will be described with reference to the schematic diagram of FIG. 7, which shows specific examples of each desired type of division and combination of subtitle text. In the following description, the user interface unit 25 is implemented as a touch panel, but is not limited to this. This also applies to the following embodiments. An example of division at times t1 to t3 and an example of combination at times t4 to t6 will be shown.

[0086] (time t1) At time t1, the user interface unit 25 displays two lines of subtitle text on the screen: "Reservations for large-scale vaccination centers for the new coronavirus vaccine start today." At this time, it is assumed that the user performs a slash action UA1 by touching and dragging the finger from the upper right to the lower left on the part of the displayed subtitle text, "starting today."

[0087] The user interface unit 25 detects the slash action UA1, acquires the start and end positions of the slash action UA1, and generates action data including the start and end positions (step ST8-1). Thereafter, the user interface unit 25 notifies the correction request receiving unit 26 of the action data.

[0088] The correction request receiving unit 26 receives the action data (step ST8-2) and determines whether it is a slash action based on the start and end positions of the action data (step ST8-3). Specifically, if the start position of the action data is in the upper right and the end position is in the lower left, it is determined to be a slash action.

[0089] If the correction request receiving unit 26 determines that the action is a slash action, it determines the range of the subtitle text to be corrected from the display position of the subtitle text and the vicinity of the start and end positions of the action data. For example, the correction request receiving unit 26 calculates the center position from the start and end positions of the action data, finds the character position of the subtitle text closest to the center position, and sets several characters before and after that character position as the range of the correction. Furthermore, if the correction request receiving unit 26 determines that the action is a slash action, it sets the request type to a split type and transmits the request type and the range of the correction to the regeneration unit 24b.

[0090] If the desired type is the division type, the regeneration unit 24b estimates a likely division position within the correction range based on the acquired correction range and the history data 21a, and regenerates the subtitle text (step ST8-4).

[0091] Specifically, the regeneration unit 24b calculates the segmentation estimation score for each word for the acquired correction range "The reservation for starts today" in the same manner as described above, and estimates the position within the correction range with the highest segmentation estimation score, excluding the currently segmented position, as the segmentation position. In this example, the segmentation position is estimated to be before the subtitle text "starting today." If the number of lines after segmentation (3 lines) exceeds the number of lines (2 lines) that can be displayed on one screen, the regeneration unit 24b deletes the excess portion ("starting today" on the third line) from the currently displayed subtitle text and creates update information for the history data 21a. Furthermore, although not this example, if the number of characters after segmentation exceeds the number of characters that can be displayed on one line, the regeneration unit 24b deletes the excess portion from the line and creates update information for the history data 21a. For example, if part of the first line of subtitle text is segmented and moved to the second line among two lines of subtitle text currently being displayed, the number of characters in the second line of subtitle text may exceed the number of characters that can be displayed on one line. In such cases, the excess portion can be deleted from the second line of subtitle text.

[0092] Furthermore, the regeneration unit 24b regenerates the subtitle text from the history data 21a corresponding to the currently displayed subtitle text "Reservations for the large-scale vaccination center for the COVID-19 vaccine start today" based on the estimated division position (before "starting today"). In this case, the regeneration unit 24b regenerates the subtitle text "Reservations for the large-scale vaccination center for the COVID-19 vaccine start today."

[0093] (time t2) Thereafter, the user interface unit 25 presents the regenerated subtitle text "Reservations made at large-scale vaccination centers for COVID-19 vaccines" to the user (step ST8-5).

[0094] The history update unit 23 updates the history data 21a stored in the history unit 21 based on the regenerated subtitle text (step ST8-6). For example, the history update unit 23 updates the history data array based on the update information of the history data 21a.

[0095] (time t3) After the display of the subtitle text displayed at time t2 ends, the user interface unit 25 presents the newly generated subtitle text "Starting today" to the user. The subtitle text shown at time t3 includes the subtitle text "Starting today" at the beginning, which was deleted following the slash action UA1 at time t1. If the slash action UA1 does not occur at time t3p, the subtitle text "Starting today" that is presented at time t3p would be the subtitle text "Starting today" that does not include the opening subtitle text "Starting today."

[0096] The above is an example of dividing the subtitle text at times t1 to t3. Next, an example of joining the subtitle text at times t4 to t6 will be described.

[0097] (time t4) At time t4, the user interface unit 25 displays two lines of subtitle text on the screen: "Vaccinations will be administered at the Otemachi Joint Government Building in Chiyoda Ward, Tokyo, which will be the Tokyo venue." At this time, it is assumed that the user performs a swipe action UA2 by touching and sliding their finger from right to left in an empty space to the right of a portion of the displayed subtitle text, "...in Chiyoda Ward."

[0098] The user interface unit 25 detects the swipe action UA2, acquires the start position and end position of the swipe action UA2, and generates action data including the start position and end position (step ST8-1). Thereafter, the user interface unit 25 notifies the correction request receiving unit 26 of the action data.

[0099] The correction request receiving unit 26 receives the action data (step ST8-2) and determines whether it is a swipe action based on the start and end positions of the action data (step ST8-3). Specifically, if the height of the start position and the height of the end position of the action data are approximately the same and the start position is to the right of the end position, it is determined to be a swipe action.

[0100] If the action is determined to be a swipe action, the correction request receiving unit 26 determines the range of subtitle text to be corrected based on the display position of the subtitle text and the start position of the action data. For example, the correction request receiving unit 26 determines the range of correction to be several characters before and after the character position of the subtitle text closest to the start position of the action data. Furthermore, if the action is determined to be a swipe action, the correction request receiving unit 26 sets the request type to a combined type and notifies the regeneration unit 24b of this together with the range of correction.

[0101] If the desired type is the combination type, the regenerator 24b estimates the combination position and the division position after combination based on the acquired correction target range and the history data 21a, and regenerates the subtitle text (step ST8-4).

[0102] Specifically, the regeneration unit 24b re-estimates the current division position included in the history data 21a corresponding to the acquired correction target range, "At the Otemachi Joint Government Building in Chiyoda Ward, which is the setting," as the joining position. Furthermore, the regeneration unit 24b calculates the division estimation score for each word for the correction target range, "At the Otemachi Joint Government Building in Chiyoda Ward, which is the setting," as described above, and re-estimates the position with the highest division estimation score within the correction target range, excluding the current division position, as the post-joining division position. In this example, the position before the subtitle text "So" is re-estimated as the division position. Note that, although not in this example, if the number of characters after division exceeds the number of characters that can be displayed in one line, the excess portion is deleted from the line, and update information for the history data 21a is created. For example, assume that the number of characters in the first line of subtitle text among two lines currently being displayed is close to the number of characters that can be displayed in one line. In this case, when the segmentation estimation score for each word in the range to be corrected is calculated according to the desired join type for the first line of subtitle text, the number of characters in the first line of subtitle text at the estimated segmentation position may exceed the number of characters that can be displayed on one line. In such cases, the excess portion can be deleted from the first line of subtitle text and moved to the second line of subtitle text.

[0103] Thereafter, the regeneration unit 24b combines subtitle text from the history data 21a corresponding to the currently displayed subtitle text "Vaccination will be carried out at the Otemachi Joint Government Building in Chiyoda Ward, which will be the Tokyo venue," based on the re-estimated combining position.

[0104] Next, the regeneration unit 24b regenerates the subtitle text from the history data 21a corresponding to the combined subtitle text "Vaccination will be held at the Otemachi Joint Government Building in Chiyoda Ward, which will be the Tokyo venue," based on the re-estimated division position (before "then"). In this case, the regeneration unit 24b regenerates the subtitle text "Vaccination will be held at the Otemachi Joint Government Building in Chiyoda Ward, which will be the Tokyo venue."

[0105] (time t5) Thereafter, the user interface unit 25 presents the regenerated subtitle text "Vaccination will be held at the Otemachi Joint Government Building in Chiyoda Ward, which will be the Tokyo venue" to the user (step ST8-5).

[0106] The history update unit 23 updates the history data 21a stored in the history unit 21 based on the regenerated subtitle text (step ST8-6). For example, the history update unit 23 updates the history data array based on the update information of the history data 21a.

[0107] (time t6) After the display of the subtitle text displayed at time t5 has ended, the user interface unit 25 presents the newly generated subtitle text "Nurses and members of the Self-Defense Forces held a ceremony to mark the completion of their formation" to the user.

[0108] The above is an example of combining subtitle texts at times t4 to t6. In this way, subtitle texts can be divided or combined depending on the user's actions.

[0109] As described above, according to the second embodiment, the user interface unit 25 serving as a detection unit detects actions taken by the user on portions of the presented subtitle text and generates action data. The correction request receiving unit 26 serving as a determination unit determines the range of subtitle text to be corrected and the desired type of subtitle text, either division or joining, based on the action data. The regeneration unit 24b re-estimates the division or joining positions of the subtitle text based on the range of correction, the desired type, and the history data, and regenerates the subtitle text from the re-estimated results. Therefore, in addition to the effects of the first embodiment, the division and joining positions of the presented subtitle text can be easily corrected in response to user actions.

[0110] (Modification of the second embodiment) In the second embodiment, a dragging operation from the upper right to the lower left is detected as the slash action UA1, but this is not limiting. For example, a dragging operation from the upper left to the lower right may be detected as the slash action UA1. Also, for example, a dragging operation straight from top to bottom may be detected as the slash action UA1. In other words, any action that indicates a division position can be used as the slash action UA1. Furthermore, other terms may be used instead of the term "slash action" as long as the action indicates a division position.

[0111] Furthermore, in the second embodiment, an operation of sliding from right to left is detected as the swipe action UA2, but this is not limiting. For example, an operation of flicking from right to left may be detected as the swipe action UA2. Also, for example, an operation of narrowing two fingers that are spread apart from each other (pinch-in) may be detected as the swipe action UA2. In other words, any action that indicates a joining position can be used as the swipe action UA2. Furthermore, any other term may be used instead of the term "swipe action" as long as the action indicates a joining position.

[0112] (Third embodiment) Compared to the second embodiment, the third embodiment adds a configuration for presenting correction candidates for subtitle text in accordance with a user action on a subtitle text portion displayed on the user interface unit 25. However, the third embodiment is not limited to this, and the configuration may be added to the first embodiment.

[0113] FIG. 8 is a block diagram showing a subtitle production device according to the third embodiment.

[0114] In the subtitle production device 20 shown in FIG. 8, compared to the configuration shown in FIG. 5, the subtitle processing unit 24 further includes a correction candidate generating unit 24c.

[0115] Accordingly, the user interface unit 25 detects an action performed by the user on a part of the presented subtitle text and generates action data, as described above. This user interface unit 25 is an example of a detection unit.

[0116] The correction request receiving unit 26 determines the range of correction to be made to the subtitle text based on the action data. Here, the correction request receiving unit 26 executes the determination when it receives action data as a correction request, for example. The correction request receiving unit 26 is an example of a determination unit.

[0117] The correction candidate generating unit 24c generates a correction candidate for the correction target range from the correction target range and the history data 21a, and estimates the division position of the subtitle text including the correction candidate.

[0118] The operation of the caption production device configured as above will be described with reference to the flowchart of Fig. 9 and the schematic diagram of Fig. 10. Note that the operation of this embodiment is executed as step ST8A between the above-mentioned step ST7 and step ST9. Step ST8A is executed as the following steps ST8A-1 to ST8A-7 and ST8-3 to ST8-6.

[0119] (Steps ST8A-1 to ST8-2) Steps ST8A-1 to ST8A-2 are executed in the same manner as steps ST8-1 to ST8-2 described above. That is, the user interface unit 25 detects an action performed by the user on a portion of the displayed subtitle text and generates action data. The correction request receiving unit 26 receives the action data from the user interface unit 25.

[0120] (Step ST8A-2-1) The correction request receiving unit 26 determines, based on the action data, whether the action data is an instruction to present correction candidates. If not, steps ST8-3 to ST8-6 described in the second embodiment are executed, step ST8A is terminated, and the process proceeds to step ST9. On the other hand, if the action data is an instruction to present correction candidates, the process proceeds to step ST8A-3. Note that this step ST8A-2-1 is omitted if the processing of the second embodiment is not executable.

[0121] (Step ST8A-3) Next, the correction request receiving unit 26 determines the range of the subtitle text to be corrected based on the action data, and notifies the correction candidate generating unit 24c of the range of the correction to be corrected.

[0122] (Step ST8A-4) The correction candidate generating unit 24c generates a correction candidate for the correction range based on the correction range of the subtitle text and the history data 21a.

[0123] (Step ST8A-5) For each generated correction candidate, the correction candidate generation unit 24c estimates the division and joining positions of the subtitle text when the correction candidate is selected, and generates candidate subtitle text based on the division and joining positions. Thereafter, the correction candidate generation unit 24c transmits the correction candidates and candidate subtitle text to the user interface unit 25. At this time, one or more correction candidates and candidate subtitle text corresponding to each correction candidate are transmitted.

[0124] (Step ST8A-6) The user interface unit 25 presents the generated one or more correction suggestions to the user.

[0125] (Step ST8A-7) When one of the correction candidates is selected in response to a user action, the user interface unit 25 presents the candidate subtitle text corresponding to the selected correction candidate to the user. Thereafter, the history update unit 23 updates the history data 21a stored in the history unit 21 based on the presented candidate subtitle text. As a result, the subtitle production device 20 ends step ST8A and proceeds to step ST9 described above.

[0126] Next, the above-described operation will be described with reference to the schematic diagram of Fig. 10, showing a specific example of presenting correction candidates for subtitle text and correcting them. In the following explanation, an example of correcting subtitle text from time t6 to t8 will be shown.

[0127] (time t6) At time t6, the user interface unit 25 displays on the screen two lines of subtitle text: "The Self-Defense Forces' nurses held a ceremony to conclude their formation." At this time, it is assumed that the user performs a pointing action UA3 by pressing and releasing their finger on the character "の" in the displayed subtitle text.

[0128] The user interface unit 25 detects the pointing action UA3, acquires the pointing position on the screen due to the pointing action UA3, and generates action data including the pointing position (step ST8A-1). Thereafter, the user interface unit 25 notifies the correction request receiving unit 26 of the action data.

[0129] The correction request receiving unit 26 receives the action data (step ST8A-2) and determines whether the action data is an instruction to present a correction candidate (pointing action) based on the pointing position of the action data (step ST8A-2-1). Specifically, since the pointing position of the action data is a single point, it is determined to be a pointing action. In other words, since the start position and end position of the action data are the same, it is determined to be a pointing action.

[0130] If it is determined that the action is a pointing action, the correction request receiving unit 26 determines the range of the subtitle text to be corrected based on the action data, and notifies the correction candidate generating unit 24c of the range of the correction to be corrected.

[0131] For example, the correction request receiving unit 26 estimates the section with the highest error rate around the pointing position pointing to "no" and sets the character string "noikan ya" in that section as the correction target range (step ST8A-3). However, this is not limiting, and the correction request receiving unit 26 may set several characters before and after the character position of the subtitle text displayed closest to the pointing position as the correction target range. Furthermore, if the correction request receiving unit 26 determines that the action is a pointing action, it transmits the correction target range "noikan ya" to the correction candidate generating unit 24c.

[0132] Next, the correction candidate generation unit 24c acquires the history data 21a corresponding to the subtitle text being displayed from the history unit 21. The correction candidate generation unit 24c also masks the portion of the subtitle text being displayed that is to be the correction range, and then uses the pre-training model BERT to predict (generate) candidate character strings (correction candidates) for the correction range (step ST8A-4). Here, BERT is an abbreviation for "Bidirectional Encoder Representations from Transformer." In this example, the correction candidate generation unit 24c predicts "noikan ya" (replace with "ikan"), "no" (delete "ikan ya"), and "no, ikan ya" (insert ",") as correction candidates for the correction range "noikan ya." Furthermore, the correction candidate generation unit 24c estimates the division position of the subtitle text including three correction candidates: "Medical officers and nursing officers of the Self-Defense Forces held a formation completion ceremony," "Nursing officers of the Self-Defense Forces held a formation completion ceremony," and "Nursing officers of the Self-Defense Forces held a formation completion ceremony." In this case, the division position is before "formation completion ceremony," which is the same as before the correction.

[0133] Next, the correction candidate generation unit 24c generates first to third candidate subtitle texts according to the three correction candidates and the division positions (step ST8A-5). The first candidate subtitle text is "Medical officers and nursing officers of the Self-Defense Forces held a ceremony to conclude the ¥n formation." The second candidate subtitle text is "Nursing officers of the Self-Defense Forces held a ceremony to conclude the ¥n formation." The third candidate subtitle text is "Medical officers and nursing officers of the Self-Defense Forces held a ceremony to conclude the ¥n formation." The correction candidate generation unit 24c also transmits the correction candidates and candidate subtitle texts to the user interface unit 25.

[0134] (time t7) The user interface unit 25 presents the generated three correction candidates to the user (step ST8A-6).

[0135] (time t8) When the user selects the correction candidate "medical officer" (replaced with "medical officer") in response to an action by the user, the user interface unit 25 presents the user with candidate subtitle text corresponding to the correction candidate. In this example, the first candidate subtitle text "Medical officers and nursing officers of the Self-Defense Forces held a ceremony to conclude their formation" is displayed.

[0136] The above is an example of correcting the subtitle text from time t6 to time t8. In this way, the subtitle text can be corrected in response to the user's actions.

[0137] As described above, according to the third embodiment, the user interface unit 25 serving as a detection unit detects actions taken by the user on portions of presented subtitle text and generates action data. The correction request receiving unit 26 serving as a determination unit determines a range of subtitle text to be corrected based on the action data. The correction candidate generation unit generates correction candidates for the range of subtitle text to be corrected based on the range of correction and history data 21a, and estimates the division positions of the subtitle text including the correction candidates. Therefore, in addition to the effects of the first embodiment, correction candidates for presented subtitle text can be easily generated in response to user actions. Furthermore, by selecting correction candidates in response to user actions, subtitle text can be easily corrected.

[0138] <Fourth embodiment> 11 is a block diagram illustrating a hardware configuration of a subtitle production device according to a fourth embodiment. The fourth embodiment is a specific example of the first to third embodiments, in which the subtitle production device 20 is realized by a computer.

[0139] The subtitle generation device 20 includes, as hardware, a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a program memory 203, an auxiliary storage device 204, and an input / output interface 205. The CPU 201 communicates with the RAM 202, the program memory 203, the auxiliary storage device 204, and the input / output interface 205 via a bus. That is, the subtitle generation device 20 of this embodiment is realized by a computer having such a hardware configuration.

[0140] The CPU 201 is an example of a general-purpose processor. The RAM 202 is used by the CPU 201 as a working memory. The RAM 202 includes a volatile memory such as a Synchronous Dynamic Random Access Memory (SDRAM). The RAM 202 may also be used as a history unit 21 that stores history data. The program memory 203 stores a subtitle generation program for implementing each unit according to each embodiment. This subtitle generation program may be, for example, a program that causes a computer to function as each of the aforementioned units, such as the history unit 21, acquisition unit 22, history update unit 23, subtitle processing unit 24, and user interface unit 25. The program memory 203 may be, for example, a read-only memory (ROM), a part of the auxiliary storage device 204, or a combination thereof. The auxiliary storage device 204 non-temporarily stores data. The auxiliary storage device 204 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). It should be noted that the RAM 202, the program memory 203, and the auxiliary storage device 204 are not limited to being built into the computer, and may be externally attached to the computer.

[0141] The input / output interface 205 is an interface for connecting to other devices, and is used to connect to, for example, a keyboard, a mouse, and a display.

[0142] The subtitle generation program stored in the program memory 203 includes computer-executable instructions. When executed by the CPU 201, which is a processing circuit, the subtitle generation program (computer-executable instructions) causes the CPU 201 to perform predetermined processing. For example, when executed by the CPU 201, the subtitle generation program causes the CPU 201 to perform the series of processing described with reference to the respective units in FIGS. 1, 5, and 8. For example, when executed by the CPU 201, the computer-executable instructions included in the subtitle generation program cause the CPU 201 to perform a subtitle generation method. The subtitle generation method may include steps corresponding to the functions of the history unit 21, acquisition unit 22, history update unit 23, subtitle processing unit 24, generation unit 24a, and user interface unit 25 described above. Here, the subtitle processing unit 24 may further include a regeneration unit 24b and a correction candidate generation unit 24c in addition to the generation unit 24a. The subtitle generation method may also include steps corresponding to the functions of the correction request receiving unit 26. The subtitle generation method may also include the steps shown in FIGS. 3, 6, and 9 as appropriate.

[0143] The subtitle generation program may be stored in a computer-readable storage medium and provided to the subtitle generation device 20, which is a computer. In this case, for example, the subtitle generation device 20 may further include a drive (not shown) for reading data from the storage medium and acquire the subtitle generation program from the storage medium. Examples of storage media that can be used include magnetic disks, optical disks (CD-ROM, CD-R, CD-RW, DVD-RAM, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), and semiconductor memories. The storage medium may also be referred to as a non-transitory computer-readable storage medium. Alternatively, the subtitle generation program may be stored in a server on a communication network, and the subtitle generation device 20 may download the subtitle generation program from the server using the input / output interface 205.

[0144] The processing circuit that executes the subtitle generation program is not limited to a general-purpose hardware processor such as CPU 201, but may also be a dedicated hardware processor such as an ASIC (Application Specific Integrated Circuit). The term processing circuit (processing unit) includes at least one general-purpose hardware processor, at least one dedicated hardware processor, or a combination of at least one general-purpose hardware processor and at least one dedicated hardware processor. In the example shown in Figure 11, CPU 201, RAM 202, and program memory 203 correspond to the processing circuit.

[0145] According to at least one of the embodiments described above, when generating subtitles from speech recognition results in real time, it is possible to reduce the burden of correcting subtitles to make them easier to read.

[0146] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as defined in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0147] 10...speech recognition unit, 20...subtitle generation device, 21...history unit, 21a...history data, 22...acquisition unit, 23...history update unit, 24...subtitle processing unit, 24a...generation unit, 24b...regeneration unit, 24c...correction candidate generation unit, 25...user interface unit, 26...correction request receiving unit, 201...CPU, 202...RAM, 203...program memory, 204...auxiliary storage device, 205...input / output interface, UA1...slash action, UA2...swipe action, UA3...pointing action.

Claims

1. an acquisition unit that sequentially acquires text of the speech recognition result; a history unit that stores, as history data, the text, an identification ID of the text, and the number of characters already output for subtitle text generated from the text; a generation unit that, in parallel with the processing by the acquisition unit and the history unit, estimates division positions and combination positions of the text from the one or more pieces of history data that have been saved, combines the text at the combination positions to create concatenated text, divides the concatenated text at the division positions to generate subtitle text, which is the concatenated text before the division positions, and generates update information that includes the identification ID of text included in the subtitle text and the number of output characters related to the text; a history update unit that updates, in parallel with the processing by the acquisition unit and the history unit, the number of output characters of the history data having an identification ID that matches an identification ID in the update information based on the update information, to the number of output characters in the update information; a presentation unit that presents the subtitle text in parallel with the processing by the acquisition unit and the history unit; Equipped with the generation unit, in parallel with the processing by the acquisition unit and the history unit, determines whether or not there is unpresented history data including text that is not presented as the subtitle text, and if there is unpresented history data, repeatedly executes the processing by the generation unit, the history update unit, and the presentation unit, and if it is determined that there is no unpresented history data, ends the processing. Subtitle generation device.

2. the generation unit estimates the joining positions so as to join the texts of the history data until the number of characters of the concatenated text obtained by joining the texts becomes equal to or greater than a threshold number of characters; The subtitle generating device according to claim 1 .

3. the history data includes the text and the date and time of acquisition of the text; the generation unit estimates the combining position so as to combine text of history data that has not been displayed even after a certain time has elapsed since the acquisition date and time; The subtitle generating device according to claim 1 .

4. the generation unit performs either one or both of morphological analysis and dependency analysis on the concatenated text obtained by combining the texts, generates a feature vector for each word based on the execution result, calculates a division estimation score for each word using the feature vector, and estimates the division position from the division estimation score; The subtitle generating device according to claim 2 or 3.

5. the generation unit performs either one or both of morphological analysis and dependency analysis on the concatenated text obtained by combining the texts, aggregates the results of the analysis on a phrase-by-phrase basis to generate a feature vector for each phrase, calculates a segmentation estimation score for each phrase using the feature vector, and estimates a segmentation position from the segmentation estimation score. The subtitle generating device according to claim 2 or 3.

6. the generation unit performs either one or both of morphological analysis and dependency analysis on the concatenated text obtained by combining the texts, generates an embedding vector for each word based on the execution result, calculates a division estimation score for each word using the embedding vector, and estimates the division position from the division estimation score. The subtitle generating device according to claim 2 or 3.

7. When calculating the segmentation estimation score for each word, the generation unit calculates the segmentation estimation score using an LSTM model that takes into account a context before each word, a context after each word, or a context before and after each word.

7. The subtitle generating device according to claim 4 or 6.

8. When the number of pieces of history data including text that is not presented as the subtitle text among the history data stored in the history unit is less than a threshold, the generation unit estimates the division position by calculating the division estimation score. The subtitle generating device according to any one of claims 4 to 7.

9. the generation unit estimates the division positions for a concatenated text obtained by combining the texts according to a fixed number of characters different from the threshold number of characters; The subtitle generating device according to claim 2.

10. when a number of pieces of history data including text that is not presented as the subtitle text among the history data stored in the history unit is equal to or greater than a threshold, the generation unit estimates the division position according to the fixed number of characters. The subtitle generating device according to claim 9.

11. a detection unit that detects an action performed by a user on a portion of the presented subtitle text and generates action data; a determination unit that determines a range to be corrected for the subtitle text and a desired type of division or combination for the subtitle text based on the action data; a regeneration unit that re-estimates division or joining positions of the subtitle text based on the correction target range, the request type, and the history data, and re-generates the subtitle text from a re-estimated result. The subtitle generating device according to any one of claims 1 to 10.

12. a user interface unit that generates action data representing an action performed by a user on a portion of the presented subtitle text; a determination unit that determines a correction target range of the subtitle text based on the action data; a correction candidate generation unit that generates correction candidates for the correction range from the correction range and the history data, and estimates division positions of the subtitle text including the correction candidates; The subtitle production device according to claim 1 , further comprising:

13. A subtitle generation method executed by a subtitle generation device including an acquisition unit, a history unit, a generation unit, a history update unit, and a presentation unit, comprising: the acquiring unit sequentially acquires text of the speech recognition result; the history unit stores, as history data, the text, an identification ID of the text, and the number of characters already output for subtitle text generated from the text; the generation unit, in parallel with the processing by the acquisition unit and the history unit, estimates division positions and combination positions of the text from the one or more pieces of saved history data, combines the text at the combination positions to create concatenated text, divides the concatenated text at the division positions to generate subtitle text, which is the concatenated text before the division positions, and generates update information including the identification ID of text included in the subtitle text and the number of output characters related to the text; the history update unit, in parallel with the processing by the acquisition unit and the history unit, updates, based on the update information, the number of output characters of the history data having an identification ID that matches an identification ID in the update information, to the number of output characters in the update information; the presentation unit presents the subtitle text in parallel with the processing by the acquisition unit and the history unit; the generation unit, in parallel with the processing by the acquisition unit and the history unit, determines whether or not there is unpresented history data including text that is not presented as the subtitle text, and if there is unpresented history data, repeatedly executes the processing by the generation unit, the history update unit, and the presentation unit, and if it is determined that there is no unpresented history data, terminates the processing; A subtitle generation method comprising:

14. Computer, an acquisition means for sequentially acquiring text of the speech recognition results; a history unit for storing, as history data, the text, an identification ID of the text, and the number of characters already output for the subtitle text generated from the text; a generation means for estimating, in parallel with the processing by the acquisition means and the history means, division positions and combination positions of the text from the one or more pieces of history data that have been saved, combining the text at the combination positions to create concatenated text, dividing the concatenated text at the division positions to generate subtitle text, which is the concatenated text before the division positions, and generating update information including the identification ID of text included in the subtitle text and the number of output characters related to the text; a history update means for updating, in parallel with the processing by the acquisition means and the history means, the number of output characters of the history data having an identification ID that matches an identification ID in the update information based on the update information, to the number of output characters in the update information; a presentation means for presenting the subtitle text in parallel with the processing by the acquisition means and the history means; A subtitle generation program for causing a subtitle to function as a the generating means, in parallel with the processing by the acquiring means and the history means, determines whether or not there is unpresented history data including text that has not been presented as the subtitle text, and if there is unpresented history data, repeatedly executes the processing by the generating means, the history updating means, and the presenting means, and if it is determined that there is no unpresented history data, terminates the processing; a subtitle generation program that further causes the computer to function as a subtitle generator;

Images