Providing subtitles for video content

Abstract

The present disclosure relates to a system and method for providing subtitles for a video. Audio of a video is transcribed to obtain caption text for the video. A first machine-trained model identifies sentences in the caption text. A second model identifies sentence breaks in the sentences identified using the first machine-trained model. Based on the identified sentences and sentence breaks, one or more words in the caption text are grouped into clip captions that are displayed for a corresponding clip of the video.

Description

[Technical Field]

[0001] This application relates to a system and method for providing subtitles to a video. [Overview of the project]

[0002] One aspect of the present disclosure provides a method for providing subtitles to a video. The method includes the steps of: processing audio data of a video to generate a timed script in a first language including a first word sequence and a timestamp for each word in the first word sequence; processing the first word sequence and using a first machine-trained model to calculate the sentence-end probability of each word in the first word sequence; determining, based on the sentence-end probability of the first word, that the first word in the first sequence is a first sentence-end word and defining a first sentence ending with the first word; and processing the first sentence and using a second machine-trained model to determine that at least one word in the first sentence is The method comprises one or more of the following steps: calculating the probability of a sentence segmentation; determining the second word of the first sentence as the clip-ending word based on the probability of a second word, and defining a first clip text ending with the second word, wherein the definition of the first clip text further defines a first clip duration corresponding to the first clip text and ending when the second word is spoken in the video; and generating subtitle data in a first language, which includes the first clip text and information indicating a first clip duration in which the first clip text is displayed as a subtitle in a first language.

[0003] In an embodiment, the method further includes determining the third word of the first sentence as another clip-ending word based on the in-sentence segmentation probability of the third word, thereby defining a second clip text that begins with the word immediately following the second word and ends with the third word, and the definition of the second clip text further defines a second clip corresponding to the second clip text that ends when the third word is spoken in the video.

[0004] In the embodiment, the timed script does not include punctuation that indicates the end of the first sentence, or an intra-sentence delimiter in the first sentence.

[0005] In one embodiment, the first machine-trained model is trained using multiple punctuated texts, each containing one or more sentence-ending punctuation marks, and is configured to calculate the probability that at least one sentence-ending punctuation mark immediately follows at least one word in the input text.

[0006] In one embodiment, a second machine-trained model is trained using multiple punctuated texts, each containing one or more sentence-delimiting punctuation marks, and is configured to calculate the probability that at least one sentence-delimiting punctuation mark immediately follows at least one word in the input text.

[0007] In the embodiment, at least one sentence-ending punctuation mark includes one of a period, a question mark, an exclamation mark, and an ellipsis, and at least one sentence-delimiting punctuation mark includes one of a comma, a colon, a semicolon, and an ellipsis.

[0008] In this method, processing the audio data of a video to generate a timed script may include performing speech-to-text (STT) processing on the audio data, in which the audio corresponding to a second word is transcribed to the second word, the time at which the second word was spoken in the video is determined, and the time at which the second word was spoken is specified in the timed script for the second word. Information indicating the first clip period may include the time at which the second word was spoken, as determined by the STT processing. Generating subtitle data in a first language may include associating the time at which the second word was spoken, as determined by the STT processing, with the first clip text as the end of the first clip period, according to a predetermined subtitle file format.

[0009] In this method, the step of processing the audio data of a video to generate a timed script may include one or more of the following steps: identifying silence and non-silence in the audio data, wherein non-silence sounds include the corresponding sounds of a second word; transcribing the corresponding sounds of a second word to a second word to obtain a first word sequence; determining the end time at which the corresponding sound of a second word ends in the video for a second word; and including the determined end time as a timestamp for the second word in the timed script.

[0010] In this method, the step of processing the audio data of a video to generate a timed script may include one or more of the following steps: obtaining a prewritten script of the video that includes a first word sequence but does not include a timestamp of the first word sequence; identifying the corresponding sound in the audio data that identifies the first sound corresponding to a second word for each word in the first word sequence; determining the end time of the first sound when the first sound ends in the video; and generating a timed script by combining the first word sequence with the determined end time so that the determined end time is specified as the timestamp of the second word.

[0011] In this method, the timed script may include a timestamp of the second word indicating the time the second word is spoken in the video, and generating subtitle data in the first language may include specifying the timestamp of the second word as the end of the first clip period according to a predetermined subtitle format. In an embodiment, the first clip text begins with the third word of the first sentence, the timed script includes a timestamp of the third word indicating the time the audio of the third word begins in the video, and generating subtitle data in the first language includes a timestamp of the third word as the start of the first clip period according to a predetermined subtitle format.

[0012] In this embodiment, the subtitle data in the first language is configured such that the entire first clip text is displayed as a subtitle for the video at the start of the first clip period and is maintained without interruption until the end of the first clip period.

[0013] In this embodiment, the first clip text further includes a fourth word between the third and second words, and the subtitle data in the first language does not include a timestamp for the fourth word, so that the first clip text is displayed as a subtitle without referring to the fourth word.

[0014] In this method, the information indicating the first clip duration may include a first timestamp indicating the start time of the first clip in the video, and may further include a second timestamp indicating the end time of the first clip in the video, and the first clip text may be displayed together with the video without interruption from the start time of the first clip to the end time of the first clip.

[0015] In this method, the timestamp for each word can define the time when the sound of the word ends in the video. The timestamp for each word can define the time when the sound of the word begins in the video.

[0016] In embodiments, the method may further include one or more of the following steps: translating a first sentence into a first translated sentence in a second language, wherein the first translated sentence ends with a first translated word; processing the first translated sentence and using a third machine-trained model to calculate the intermolecular delimitation probability of at least one word in the first translated sentence; determining a second translated word in the first translated sentence as a clip-ending word based on the intermolecular delimitation probability of the second translated word, thereby defining a first translated clip text ending with the second translated word; and generating second language subtitle data including the first translated clip text and information indicating the second language period in which the first translated clip text is displayed as a subtitle in the second language. In the embodiment, the first clipping period for displaying the first clipping text is identical or substantially identical to the second language period for displaying the first translated clipping text, regardless of whether the second word ending the first clipping text semantically corresponds to the second translated word ending the first translated clipping text.

[0017] In the embodiment, the first translated clip text in the second language may not have a semantic correspondence to the first clip text in the first language. The first translated clip text may be a translation of the first clip text.

[0018] In one embodiment, generating second language subtitle data includes specifying the time when the second word is spoken in the first language as the end of the second language period, such that the first clip period and the second language period end simultaneously.

[0019] In the embodiment, the timed script includes a timestamp of the second word indicating the time the second word was spoken in the video, and generating subtitle data in the second language includes specifying the timestamp of the second word as the end of the first clip period and specifying the second language period in the second language subtitle data according to a predetermined subtitle format.

[0020] In an embodiment, the first clip text starts with the third word of the first sentence, the first translated clip text starts with the third translated word of the first translated sentence, the timed script includes a time stamp of the third word indicating the time when the sound of the third word starts in the video, and generating the subtitle data in the second language further includes designating, in the subtitle data in the second language, the time stamp of the third word as the start of the first clip period and the second language period, and the first clip period and the second language period are the same regardless of whether the third word semantically corresponds to the third translated word.

[0021] In an embodiment, the first translated sentence does not include punctuation marks indicating in-sentence delimiters in the first translated sentence, the third machine-trained model is trained using a plurality of punctuated sentences in the second language, and is configured to calculate the probability that at least one in-sentence delimiter punctuation mark follows at least one word in the input sentence.

[0022] In an embodiment, when "n" is a natural number greater than "2", the first sentence is divided into "n" clip texts based on at least the first word and the second word, and the first translated sentence is divided into the same number "n" of translated clip texts.

[0023] In an embodiment, the method further includes one of determining the third word of the first sentence as the clip end word, and a second clip text is defined that starts with a word immediately following the second word and ends with the third word. In an embodiment, defining the second clip text further defines a second clip corresponding to the second clip text, ends when the third word is spoken in the video, and the third word of the first sentence is identified as the clip end word based on at least one of the in-sentence delimiter probability of the third word and the length of the silence following the sound of the third word in the video.

[0024] In an embodiment, the subtitle data in the first language is configured such that the entire first clip text is displayed as subtitles of the video at the start of the first clip period and is maintained without interruption until the end of the first clip period. In an embodiment, the subtitle data in the second language is configured such that the entire first translated clip text is displayed as subtitles of the video at the start of the second language period and is maintained without interruption until the end of the second language period.

Brief Description of the Drawings

[0025] [Figure 1] It is a flowchart diagram of an embodiment for providing subtitles to a video. [Figure 2] It is a flowchart diagram of an embodiment for providing translated subtitles to a video. FIG. 2 presents an embodiment of a method for translating subtitles into different languages. [Figure 3] It presents a platform user interface where subtitles can be combined with video clips and edited together. [Figure 4] It is a diagrammatic representation of the association between a text file and a video when a text model is executed on the video and its text file(s). [Figure 5] It is a diagrammatic representation of an embodiment showing the association between a text file and a video when an AI model is executed on the video and its text file(s) to generate clips. [Figure 6] It is a diagrammatic representation of an embodiment showing the association between a text file in a different language and a video when an AI model is executed on the video and its text file and translated into another language. [Figure 7A] It presents an embodiment of a method for detecting sentence endings, in - sentence delimiters, and optional translations into another language to create subtitles in one or more languages for video clips. [Figure 7B] It presents an embodiment of a method for detecting sentence endings, in - sentence delimiters, and optional translations into another language to create subtitles in one or more languages for video clips. [Figure 8]This is an illustrative diagram of an exemplary machine in the form of a computer system that can be used to perform any of the methods disclosed herein. [Figure 9A] An example of transcribed text obtained from video speech-to-text (STT) processing is shown. [Figure 9B] Figure 9A shows an example of a timed script with timecodes added to the transcribed text. [Figure 10] This is an example of calculating the sentence-end probability of a word in the transcribed text shown in Figure 9A. [Figure 11] The following are example sentences identified from the transcribed text in Figure 9A. [Figure 12A] This demonstrates how to identify the clipping end word in the example sentence in Figure 11 based on the probability of sentence segmentation. [Figure 12B] This is an example of splitting the example sentence in Figure 11 into two clipped texts. [Figure 13] This is an example of subtitle data generated from the timed script in Figure 9B using the two clip texts in Figure 12B. [Figure 14] The sentence-by-sentence translation of the text in Figure 11 is shown. [Figure 15A] Figure 14 shows how to identify clipping-ending words in the translated text. [Figure 15B] This indicates that the translated text will be split into two translated clipped texts. [Figure 16] Figure 15B shows an example of generating translated subtitle data using the translated clip text.

[0026] The accompanying drawings, with similar reference numerals indicating identical or functionally similar elements across separate drawings, are incorporated herein by and form part of this specification and serve to further illustrate embodiments of the concept including the claimed disclosures, and to illustrate the various principles and advantages of those embodiments.

[0027] The methods and systems disclosed herein are represented, where appropriate, by conventional symbols in the drawings, and only specific details suitable for understanding the embodiments of this disclosure are shown so as not to obscure the disclosure by details that would be readily apparent to those skilled in the art who have an interest in the description herein. [Modes for carrying out the invention]

[0028] Embodiments of the present invention will be described below with reference to the drawings. These embodiments are provided to better understand the present invention, and the present invention is not limited to these embodiments. Changes and modifications that are apparent from those embodiments still remain within the scope of the present invention. On the other hand, the original claims constitute part of the detailed description of this application.

[0029] The need to provide video subtitles Many creators monetize their videos on platforms like YouTube. Since creators earn more the more views their videos get, reaching a larger audience is crucial. Providing subtitles is one way to attract more viewers. However, creating subtitles without automation can be time-consuming and laborious.

[0030] The technology presented here This application discloses solutions, systems, and methods for generating, processing, and presenting subtitles for a video (target video). The solutions, systems, and methods presented herein are collectively referred to herein as the “Art” or “Art of Presentation.”

[0031] Non-restrictive implementation The embodiments of this technology will be described below with reference to the drawings. This technology is not limited to the embodiments described. Any obvious changes and modifications from the embodiments described remain within the scope of this technology.

[0032] Drawings to illustrate non-restrictive examples The drawings are provided in detail to illustrate non-limiting embodiments of the technology. The drawings are illustrative and not intended to limit the technology to the illustrated embodiments.

[0033] Subtitle format Subtitles can be stored as a single file. Various subtitle formats can be used. For example, SubRip, SubViewer, Timed Text Markup Language (TTML), SBV (YouTube format), Distribution Format Exchange Profile (DFXP), and Web Video Text Track (Web VTT) can be used. In embodiments, subtitles for a target video may be stored using formats other than those exemplified, and may be stored as multiple related files.

[0034] Subtitle components In embodiments, the subtitles for a target video include at least two components: (1) the text to be displayed (collectively, the “caption text” of the target video), and (2) timing information for displaying the text (timestamp, timecode). The subtitles may include one or more additional components. For example, markup (bold, italics, underline), font, font size, spacing, and position information may be included in the subtitles. In embodiments, the term “caption text” or “caption data” refers to the entire text displayed as the subtitles for the target video.

[0035] Figure 13 shows exemplary subtitle data 1300, which includes the clip's sequence number 1312, timecodes 1314 indicating the start and end of the clip, and the clip's text 1240.

[0036] Caption text obtained from the video's audio. Video subtitles include text to visualize speech or sounds within the video. This technology may obtain such text from the processing of the video's audio. This technology may use audio recorded with the video (live recording) and audio recorded separately from the video (dubbing, narration). In embodiments, caption text can be obtained using audio that is part of the video, audio associated with the video, or audio related to the video.

[0037] Speech-to-text translation (STT) to obtain caption text. This technology allows the use of speech-to-text (STT) technology with video audio. STT technology can analyze the components of audio, remove noise from the audio, recognize one or more utterances (words) from the audio, recognize one or more languages ​​of the utterances, and transcribe the recognized utterances into text data (STT text or transcribed text) in the recognized language(s). STT processing can transcribe audio word by word or character by character to obtain a sequence of spoken words. At least a portion of the obtained STT text (or a modified version thereof) can be used as caption text to visualize the recognized speech in the video. In embodiments, different audio transcription techniques may be used. Figure 9A shows transcribed text 920 obtained by processing the audio of video 910.

[0038] Using pre-prepared scripts In this embodiment, a script or screenplay prepared for video recording can be used as text data (caption text) for the video subtitles. This technology can extract line text from a pre-prepared video script, determine the video portion (clip) corresponding to the line text, and display the line text as subtitles for the determined portion of the video. In this embodiment, text other than a script or screenplay can be used.

[0039] The STT text or pre-prepared script lacks punctuation. In the embodiment, subtitles are generated from processing punctuated text. In the case of punctuated text, the technique may perform one or more of the following: removing punctuation, checking punctuation, and identifying additional punctuation.

[0040] Language of the caption data Caption text may be in one or more languages ​​spoken in the video. With respect to audio in the video, the language spoken will hereafter be referred to as the “original language” or “first language.”

[0041] Using a combination of transcribed text and pre-prepared scripts. In embodiments, STT text (transcribed text) obtained from the video's audio, a pre-prepared script for the video, and a combination of the two may be used as text data (caption text) for the video's subtitles. For example, when creating subtitles based on a pre-prepared script for the video, the technique may use one or more words in the STT text to modify (replace), add, or delete one or more words in the script to reflect what is actually spoken in the video. In another example, when creating subtitles based on a pre-prepared script for the video, the technique may use one or more words in the script to replace, add, or delete one or more words in the STT text. For example, slang spoken in the video may be replaced or deleted in the subtitles.

[0042] Determining the timing of caption text elements To synchronize a target video with its caption text, this technology determines, calculates, or selects the timing of one or more words (components) in the caption text. In embodiments, this technology determines a start time and an end time for each word in the caption text. In embodiments, timing information may be determined for one or more components of the caption text other than words (e.g., letters, clauses, phrases, sentences, paragraphs).

[0043] Determining the timing of caption text based on matching sound timing. This technology can analyze the audio of a target video to identify silence (and / or noise), identify sounds (or utterances) separated by silence or noise, and determine the timing (start / end times) of the identified sounds. In embodiments, this technology determines the timing of one or more words in a given script (caption text) that does not have timing information. This technology can identify matching sounds in the target video based on the simulated pronunciation of the words and determine the start and / or end times of the sounds as the timing of the word(s). In embodiments, this technology determines the timing of caption text words when transcribing the audio of a target video. This technology can use the start time of an utterance as the start time of the transcribed text of the audio and the end time of an utterance as the end time of the transcribed text. In embodiments, the timing of components (letters, words, phrases) of the caption text may be determined based on the timing of the corresponding sounds of the components in the target video using processes other than those exemplified.

[0044] Timing information format In the embodiment, timing information for components of the caption text may be stored using one or more of the following: time from the start of the target video, time to the start of the target video, frame number, and a code that can indicate a specific time within the target video. In the embodiment, any data format that can indicate a point in time or segment within the target video may be used.

[0045] Timing script The caption text and associated timing information are collectively referred to below as “timing text data” or “timing script.” The timing script may be a single text file containing the word sequence (caption text) in the target video and the timing for each word. In embodiments, the timing script may be stored using a non-text file format, or may be stored using multiple files. Figure 9A shows the transcribed text 920 obtained by processing the audio of video 910. Figure 9B shows an exemplary timing script 940 in which timing information is added to each word of the transcribed text 920. In the timing script 940, the word “dream” 952 is associated with the start time 954 and end time 956 of its corresponding sound.

[0046] Adjusting the timecode of a timed script to synchronize with the video's audio. Given a script with timecodes, this technology can adjust or verify the timecodes so that the words in the script are synchronized with the corresponding sounds in the video.

[0047] Clips and clip cations This technology can process a timed script to determine the target video clip (part) to display subtitles and the corresponding text (clip cation) to display as subtitles for that clip. The term "clip" (or "video clip") refers to the portion (or duration) of video that displays (or maintains) the same subtitle text. The term "clip caption" (or "clip text") refers to the text that appears as subtitles for the corresponding clip.

[0048] The same caption is maintained throughout the clip. In one embodiment, the entire clip caption is displayed at the beginning of the clip, remains visible throughout the clip, and disappears at the end of the clip. The same clip caption (clip text) may be displayed throughout the entire clip without change or interruption. In another embodiment, visual effects or markup (bold, italics, underline) may be applied to only a portion of a single clip while maintaining the same text characters. In yet another embodiment, words within a single clip caption are displayed sequentially according to individual timing information (timecodes) so that the entire clip caption is displayed at the end of the clip. In another embodiment, the clip caption may be displayed in a different way than the example, as long as the entire clip caption is displayed at least at a point in the clip.

[0049] Clip definition using clip text timing In one embodiment, the technology first defines a clip caption, and then defines a corresponding clip based on the timing information of the determined clip caption. For example, if a clip caption is defined to have a start word and an end word, the start time (timecode) of the start word is determined as the start time of the clip, and the end time (timecode) of the end word is determined as the clip. By applying predetermined time adjustments, the start time of the clip can be determined based on the timing of the start word, and the end time of the clip can be determined based on the timing of the end word. In another embodiment, the technology can first define a clip and then define its clip caption to include all the text for the corresponding period.

[0050] Word-based clip captions In embodiments, a clip caption (a single clip) is defined to contain one or more words. A single word cannot be separated into two clips. In embodiments, a single clip contains a word fragment if only the fragment is spoken in the video, or if there is a long silence between the spoken fragment and other subsequent fragments(s). In embodiments, clip captions may be defined using higher grammatical units (phrases, clauses, sentences).

[0051] Grouping of words to define clips / clip cations In embodiments, the technology groups two or more consecutive words in caption text as a single clip caption (clip text). Words may be grouped by sentence such that two words in a single sentence are included in a single clip caption. In embodiments, two words in a sentence may be separated into two clip captions if there is a long silence between the two words or if the sentence is too long for a single clip. In embodiments, a single clip may contain words from two different sentences. In embodiments, words may be grouped using grammatical units other than sentences (phrases, clauses) or segments of caption text other than grammatical units.

[0052] Identification of grammatical units / segments In embodiments, the technology may process the caption text to identify grammatical units (words, phrases, clauses, sentences) or other segments within the caption text. In embodiments, the technology can identify grammatical units or other segments by referring to punctuation marks (periods, question marks, exclamation marks, commas, etc.) in a script given as caption text. In embodiments, the technology can determine the potential locations of punctuation marks in STT text that does not contain punctuation. Exemplary processing for identifying grammatical units or other segments in caption text is described later in this disclosure.

[0053] Machine-trained models for sentence recognition In the embodiment, a machine-trained sentence recognition model (hereinafter referred to as the "sentence model" or "sentence artificial intelligence") is used to identify one or more sentences within the caption text (caption data). The sentence model processes the caption text and identifies the beginning and / or end of one or more sentences within the caption text. In the embodiment, techniques other than machine-trained models may be used.

[0054] Sentence model input - word sequence In an embodiment, the sentence model is configured to receive a predetermined number of words (e.g., 200 words) as its input. In an embodiment, the caption text (STT text, prewritten script) is divided into several smaller word sequences to satisfy predetermined requirements for input to the sentence model. If the word sequence is shorter than a predetermined number, one or more dummy words or null values ​​may be input along with the word sequence. In an embodiment, the sentence model may be flexible to receive inputs of different sizes. In an embodiment, the input data size may be defined using units other than the number of words (e.g., the number of characters).

[0055] Word pre-screening In embodiments, certain words are removed from the input text to the sentence model. For example, articles ("a", "an", and "the") may be excluded from the input to the sentence model to be computed, since articles generally do not end a sentence. In embodiments, if the script or screenplay contains words other than line text describing scenes in the video (e.g., "laughter", "background music"), such words may be excluded from the input to the sentence model.

[0056] Sentence model output - probability of sentence end / beginning In embodiments, a sentence model is configured to calculate, for one or more words in its input text, the probability that a word is the last word in a sentence (end-of-sentence probability) and / or the probability that the word is the first word in a sentence (begin-of-sentence probability). In embodiments, the end-of-sentence probability of a word represents the probability that a particular end-of-sentence punctuation mark follows that word. In embodiments, the beginning-of-sentence probability of a word represents the probability that the word follows a particular end-of-sentence punctuation mark. In embodiments, a sentence model that calculates end-of-sentence probabilities is sometimes called a sentence model that calculates beginning-of-sentence probabilities, because the beginning-of-sentence probability of a word is the same as the beginning-of-sentence probability of the following word. According to Figure 10, sentence model 1010 calculates the end-of-sentence probability 1020 (in percentage) for each word in the input text 920. Word 1022 has a 99% end-of-sentence probability.

[0057] Probability of each punctuation mark at the end of a sentence In the embodiment, a sentence model is used to calculate multiple sentence-end probabilities for each single word, corresponding to each sentence-end punctuation mark (period, question mark, exclamation mark, or ellipsis). The technique presented here can either add these multiple sentence-end probabilities to calculate a representative sentence-end probability, or obtain the highest value among the multiple sentence-end probabilities. In the embodiment, separate sentence models may be used for different sentence-end punctuation marks.

[0058] Adjusting sentence model output In some embodiments, the sentence-end probability (or sentence-start probability) calculated by the sentence model can be adjusted based on various factors. These can include predefined default probability values ​​for the words themselves, specific adjacent words, the presence of well-known or established phrases, or adjustments to specific grammatical tools or techniques.

[0059] A predetermined threshold for determining the final word of a sentence. In the embodiment, a word is determined to be a sentence-final if its probability of being at the end of a sentence is greater than a predetermined threshold. This threshold may be specific to one or more words, common to all words, set or adjusted by the sentence model, set manually by the user, or set by the software administrator or programmer. This threshold may differ for each word and its translation, or it may be uniform across the language (the same for words and all translations). In the embodiment, sentence-final words may be determined using one or more criteria other than the predetermined threshold. In Figure 10, if this threshold is 90 percent, four words 1022, 1024, 1026, and 1028 are identified as sentence-final words.

[0060] Sentence determination by sentence-final words In the embodiment, in the caption text, the word immediately following the end of a sentence may be determined as the beginning of the next sentence. The first word in the caption text data is another beginning of a sentence. A sentence is composed of one or more words from the beginning of a sentence to the end of the sentence immediately following it. A sequence of words can be identified as a sentence, but the identified sentence may not be a grammatically complete sentence. In Figure 11, the STT text 920 is divided into five segments 1110-1150 based on four sentence endings 1022-1028. Four sentences 1110-1140 are identified. In the embodiment, the last segment 1150 is combined with the beginning port of another STT text immediately following the STT text 920 to form a complete sentence (or clip) such that the segment's beginning word "I" is used as the clipping start word.

[0061] Defining clips by sentence In the embodiment, a clip caption (clip text) and its corresponding clip may be defined to include all the words of one or more complete sentences. For example, in Figure 11, each of the four sentences in Figures 1110 to 1140 may be defined as the clip text of a single clip.

[0062] When a sentence is defined as the clip text of a single clip, the clip (clip duration) can be defined using the timing information of the sentence's start and end words. The start time (timestamp, timecode) of the start word can be used as the start time of the clip, and the end time (timestamp, timecode) of the end word can be used as the end time of the clip. For example, the first sentence 1110 in Figure 11 is used as the clip text of a single clip, with the start word "so" 982 and start time "00:00,175" 984 used as the start of the clip, and the end word "dream" 952 and end time "00:00,720" 956 used as the end of the clip. In embodiments, clips corresponding to individual sentences can be combined to form longer clips, and a clip corresponding to a single sentence can be split into two or more clips based on internal sentence breaks.

[0063] Adjusting the timing of the clip In embodiments, adjustments may be made so that the clip starts a predetermined time earlier (or later) than the first word. In embodiments, adjustments may be made so that the clip ends a predetermined time later (or earlier) than the last word. In embodiments, the start and end of the clip may be defined in ways different from the examples, as long as this does not disrupt the synchronization of the clip with its corresponding sentence(s).

[0064] Sentence delimiters for defining clip captions In embodiments, the technology can process at least a portion of the caption text to identify one or more in-sentence segments and define clip cations and their corresponding clips based on these segments. For example, one or more sentences identified using a sentence model can be further analyzed to be identified by one or more breaks within the sentences, and the clips containing the sentences can be divided using the identified in-sentence segments.

[0065] In-text model In embodiments, the technology may use a machine-trained intermelligent model (hereinafter, "intermelligent model") to identify one or more breaks within a sentence in the caption text. The intermelligent model may be configured to receive a word sequence and output, for each word in the input, the probability that an intermelligent segment follows that word, or the probability that the word precedes an intermelligent segment immediately before it (hereinafter, "intermelligent segment probability").

[0066] Input to the sentence model - Sentences identified using the sentence model In one embodiment, the in-sentence model is configured to receive one or more sentences identified using the sentence model as its input. In another embodiment, the in-sentence model is configured to receive a portion of the caption text without referring to the sentences identified using the sentence model. The in-sentence model may have a maximum number of words for its input (e.g., 50 words) and may be shorter than the maximum number of words for the sentence model (e.g., 300 words).

[0067] Excluding short sentences from the input of the in-sentence model. In the embodiment, if a sentence is shorter than a predetermined length (e.g., number of characters) allowed for a single clip, it may not be necessary to split the sentence into two or more clips, and the sentence can be excluded from the input of the in-sentence model.

[0068] Output of the in-sentence model - In-sentence delimiter probability In the embodiments, the word segmentation probability represents the probability that the word is immediately before (or immediately after) one or more sentence punctuation marks that indicate a segmentation (e.g., a comma, dashed line, ellipsis, semicolon, etc.). In the embodiments, the word segmentation probability represents the probability that the word is the last word (or first word) of a phrase or clause.

[0069] Various implementations of in-sentence model probabilities In this embodiment, the in-sentence model assigns a different probability value to each word based on the probability that each word is immediately preceding a different type of in-sentence punctuation mark. For example, there is a 70% probability that the punctuation mark following the word is a comma, an 80% probability that it is an ellipsis, and a 90% probability that it is a semicolon. The in-sentence model then selects the punctuation mark with the highest probability for that word, which in this case is a semicolon.

[0070] In the embodiment, the in-sentence model simply assigns a probability score to each word in the examined sentence, based on the probability that each word is adjacent to or immediately before a comma, regardless of what punctuation may follow the word, and assigns a single probability score to each word in the examined sentence.

[0071] Depending on the embodiment, the sentence model may select the punctuation mark with the highest probability for each word, or it may assign a punctuation probability for each word based on the punctuation mark with the highest probability (i.e., in this case, the exclamation mark). In the embodiment, the sentence model may compare different probabilities of each word being immediately before different punctuation marks, and all of these comparisons can be used for the respective different probability scores of other words in the text.

[0072] According to Figure 12A, the in-sentence model 1210 calculates the in-sentence segmentation probability 1220 (percentage) of words in the input sentence 1110. Model 1210 did not calculate the in-sentence segmentation probability for word 952 because it is the last word of the sentence.

[0073] Splitting clips defined by sentences In some embodiments, a clip defined to contain or encompass one or more sentences identified using a sentence AI may be split into two or more clips by one or more intra-sentence delimiters identified using an intra-sentence model. In certain embodiments, a clip may be defined after identifying the intra-sentence delimiters using end-of-sentence and intra-sentence delimiter timestamp information.

[0074] In embodiments, the in-sentence model determines a segment or portion of a sentence by determining the position of an in-sentence delimiter, preferably by determining the position of the word immediately preceding a comma. These sentence segments or portions may be divided by in-sentence punctuation as described above, or, in alternative embodiments, by spaces, pauses, or other decisions made by the in-sentence model.

[0075] Next, in the embodiment, these sentence delimiters that define sentence portions or segments can be used to mark the location of the sentence delimiters in the STT text, the text file, and / or the corresponding location in the video clip and / or audio file, while the clips defined by the sentences may be further timestamped and / or further divided into additional clips.

[0076] Determining sentence boundaries - threshold In the embodiment, for a word considered to be in a specific position within a sentence, such as a sentence separator word, or a word immediately preceding a sentence separator which may be defined by a sentence separator, the probability of that word being a sentence separator must satisfy or exceed a predetermined threshold. In Figure 12A, if the threshold is 90%, word 962, which has a sentence separator probability of 98%, is identified as a sentence separator word.

[0077] This threshold may be defined per word, common to all words, set or adjusted by the sentence model, or manually set by the user or by the software administrator or programmer. Furthermore, the threshold may differ depending on the language. For example, in English, a specified threshold for a word considered the last word before a sentence segment might be assigned an 85% probability of a sentence segment or a score of 85. However, in Korean, it might be set to 80% or a score of 80. This threshold may be specific to one or more words, or it may be uniform for all words throughout the language. If the probability of a comma or punctuation mark assigned to a word satisfies a predefined threshold, that word is considered by the sentence model to be the last word in a sentence segment, or, in various other embodiments, to occupy a specific position within a sentence.

[0078] Adjusting the probability value of the threshold comma / punctuation mark In the embodiment, the in-sentence model can determine or adjust the probability values ​​of punctuation. The threshold may vary depending on the word, position, or space, or it may be common to all words in the language.

[0079] Use an in-sentence delimiter as the closing word for clipping. In this embodiment, the technology uses an intra-sentence delimiter word as a clipping end word so that a sentence identified in the STT text is divided into two or more clipping texts. As shown in Figures 12A and 12B, the word "tomorrow" 962 is identified as both an intra-sentence delimiter word and a clipping end word, thereby dividing the first sentence 1110 of text 920 into two clipping texts 1240 and 1260.

[0080] Generating subtitle data from a timed script Figure 13 shows exemplary subtitle data 1300 generated based on clip-ending words in the timed script 940. Of the words in script 940, four sentence-ending words 1022-1028 identified using sentence model 1010 are used as clip-ending words, and the intra-sentence segment word 962 identified using intra-sentence model 1210 is also used as a clip-ending word. In addition, the first word of script 940, "so" 982, is used as the word that starts the clip.

[0081] According to Figure 13, the first sentence 1110 of text 920 is split into two clip texts 1240 and 1260, while the second sentence 1120 remains a single clip text. The subtitle data 1300 includes three segments 1310, 1320, and 1330, corresponding to the clip texts 1240, 1260, and 1120, respectively. The first segment 1310 of clip text 1240 defines the serial number 1312 of the clip text, and the time code 1314 defines the clip duration in video 910 where clip text 1240 is displayed as a subtitle.

[0082] Location of text delimiters The positions of identified sentence delimiters can be marked in the text or STT file, which can then be linked (i.e., via timestamp information) to the positions of the words in the video and / or associated audio (audio in the video, or dubbed audio). When the sentence model parses the full text file, it identifies the ending word of each sentence segment and marks its position in the text, and subsequently in the video / audio. Thus, the marked positions (time / frame in the video) are indicators of the end of a sentence segment, and each new sentence begins from the end of the last sentence.

[0083] Results of text recognition Once a sentence segment is identified, the text, data, or STT file may be timestamped, and the corresponding location in the video clip and accompanying audio may also be timestamped. The identified or timestamped location within the clip can then be used to further divide the clip into smaller clips. The clip initially defined by the sentence model may be further cut, marked, identified, or spliced ​​into new clips at identified locations of punctuation or specific pauses, as determined by the sentence segment model's determination of sentence segments. Thus, a clip produced by the sentence segment model's identification of sentence-ending words may contain one or more other sentences that can be identified by the sentence segment model, and the first clip may be divided into separate clips, each with a clip caption consisting of a sentence segment.

[0084] Stores timestamp information for text / internal text delimiters. In embodiments, the technology may store or mark the positions of each beginning and end word in caption text (e.g., STT text), a timed script, or separate data attached to the caption text or timed script. By doing so, the identified sentences can be linked to the corresponding portion (clip) of the target video.

[0085] In this embodiment, the technology can store or mark the position of each sentence segment. This may be the end time of the word immediately preceding the segment, or the start time of the word immediately following the segment.

[0086] Alternative predefined probabilities In many embodiments, association probability values ​​or scores are provided for each word with various punctuation marks used in the given language. For example, probability values ​​for sentence-ending punctuation marks such as periods, or sentence-internal punctuation marks indicating pauses such as commas. One or more of the AI ​​models already described, or alternative algorithms, can use these pre-provided punctuation probability values ​​for each word to determine whether a comma, period, or any other appropriate available punctuation mark should be inserted adjacent to the word. The probability of punctuation marks for each word may differ on each side adjacent to that word. However, in embodiments, only one location adjacent to each word on the side most likely to contain punctuation is considered.

[0087] Post-output adjustment using an in-text model In embodiments, the in-sentence model can generate or adjust punctuation probability values ​​for words in a text file after its initial output. The adjustment may be based on a number of factors, including, but not limited to, default settings or punctuation probabilities for each word, the presence of punctuation in the input text, the presence of identified sentence-ending words, the probability that a word is a sentence-ending word, and the assigned probability values ​​for other words and punctuation in the text. In embodiments, the technique can consider silence to adjust in-sentence segmentation probabilities. For example, a word may have a higher in-sentence segmentation probability if it is followed by a pause or silence longer than a predetermined length.

[0088] Use of sentence models and in-sentence models in sequencing In the embodiment, the sentence model is first run on the input text to determine an initial set of sentences and an initial set of clips derived from the original video, with each clip containing one complete sentence. Next, a second in-sentence model is run to enhance the output of the sentence model, identifying in-sentence delimiters within the identified sentences / clips, thereby further identifying and deriving clips that need to be split into additional clips from the already identified clips, or, in some cases, combining different clips as needed to complete sentences.

[0089] Using two separate models In this embodiment, the technology uses two separate models: one for identifying sentences (sentence model) and another for identifying (in-sentence model). The in-sentence model's task is to find the appropriate location within each sentence to further break down the sentence, and it can do this more accurately than the sentence model. The in-sentence model is an AI model trained primarily for this purpose, and because it provides already defined sentence inputs both during training and when used with input data, it may be able to find commas within sentences more accurately. Since the two AI models have different inputs, different outputs, require different training datasets, and may require different training techniques to satisfy their purposes, it may be efficient to separate the sentence model and the in-sentence model.

[0090] Text-to-text model coupling In one embodiment, the technology can train a single machine-trained model to perform the functions of a sentence model and an intra-sentence model. In another embodiment, the technology can train a sentence model and an intra-sentence model and then combine the two trained models into a single model.

[0091] Probability table In embodiments, the technology may use a static table containing multiple words and one or more predetermined probability values ​​for each word. The one or more predetermined probability values ​​for a word may include one or more of the word's end-of-sentence probability and the word's in-sentence boundary probability.

[0092] Other factors for determining sentence and internal sentence boundaries In embodiments, if a word is followed by silence (or a pause) longer than a predetermined time, the present technique may determine that the word is the end of a sentence, or it may increase the probability that the word is the end of a sentence. In embodiments, if a word is followed by silence (or a pause) longer than a predetermined time, the present technique may increase the probability that the word is an intra-sentence segment, or it may determine that an intra-sentence segment follows the word. In embodiments, if the number of sentences in the input text is determined or known, the word with the highest probability may be selected as the end of a sentence to satisfy that number. The present technique may consider one or more factors other than punctuation to identify a sentence or intra-sentence segment, and may construct a sentence model or intra-sentence model accordingly.

[0093] Clips that are too long or too short The length of the clip duration (or clip text) may indicate that it is too long and needs to be further divided, or that it is too short and needs to be combined into several clips. In embodiments, the technique uses one or more of the above AI models to divide a clip into multiple clips if the length of the clip exceeds a predetermined maximum length, or to combine a clip with other clips if the clip is less than a predetermined minimum length.

[0094] For example, an in-sentence model can be expanded into clips that appear too long to be broken down into several more sentence parts. Alternatively, a sentence or in-sentence model can be used to combine a clip with surrounding clips, whether the clip is a segment of another sentence or a complete sentence. Users can manually split clips and configure settings to split overly long clips, and the maximum clip length can be manually set by the user.

[0095] Ignoring punctuation In embodiments, one or more identifiable punctuation marks (or sentence breaks) may be ignored when defining sentences and clips. For example, ignoring punctuation marks may result in longer clips that may contain multiple punctuation marks, especially if the punctuation marks do not strongly correspond to pauses or are not strong indicators of sentence end. In embodiments, one or more identifiable punctuation marks may be ignored even if the threshold is met, if the word punctuation rate is not very high relative to the threshold.

[0096] On-screen display time limit for clip captions In some embodiments, a time limit may be set for the length for which a clip caption can be displayed on a video clip. For example, a clip caption may be limited to being displayed on a video clip for a maximum defined duration. In these examples, the caption text may be removed, or the clip may be shortened, or split into several clips. The caption may also have a minimum time limit for when it must be displayed.

[0097] Training of machine-trainable models This technology may utilize various known training techniques to obtain a machine-trained model with desired performance. In embodiments, the technology presented may utilize machine learning techniques that may include, but are not limited to, deep neural networks, autoencoders, vibrations or other types of autoencoders, and generative adversarial networks.

[0098] For example, model training is complete when, for each input data in the training dataset, the output from the model falls within a predetermined acceptable range from the corresponding desired output data (label) in the training dataset.

[0099] Dataset for training machine-trainable models To prepare a machine-trained model, this technique can develop or create a training dataset for the machine-trainable model. The training dataset contains multiple data pairs. Each pair contains input data for training the machine-trainable model and desired output data (labels) from the model in response to the input data.

[0100] Training data for sentence models In an embodiment, to train a sentence model to calculate the sentence-end probability of each word in an unpunctuated input text, the training input data may include sequences of unpunctuated words, and the corresponding training output data (desired output for the input) may be values ​​indicating each sentence-end punctuation mark (e.g., 100% for a sentence-end word, 0% for other words). Sequences of unpunctuated words can be generated by removing punctuation from appropriately punctuated text. In an embodiment, the training output data may be an indication of a specific sentence-end punctuation mark. The training dataset may be in a different format than the example.

[0101] Training data for the in-text model In an embodiment, the in-sentence model may be trained primarily on a set of properly punctuated sentences. In an embodiment, the training input data may consist of one or more complete sentences without punctuation, and the corresponding training output data may be values ​​indicating the in-sentence boundaries of the sentence (e.g., 100% for words immediately followed by in-sentence punctuation, and 0% for other words). In an embodiment, the training dataset may be configured differently from the example.

[0102] A different model from static tables In this embodiment, the machine-trained model differs from a static table of words and their corresponding probabilities in that the model can output different values ​​for the same word. In Figure 10, words 1022 and 1024 have the same text "dream" but different sentence-end probability values.

[0103] Model language dependencies In embodiments, the technology may train and configure separate versions of sentence models (and intra-sentence models) for different languages. To provide subtitles for videos recorded in a first language, the technology may need to use a first-language version of a machine-trained model. Training of a first-language model may depend primarily on a training dataset in the first language, and may use additional training datasets set up in one or more foreign languages. In embodiments, the technology may train a single model to handle two or more languages.

[0104] Video processing in Clip View In the embodiment, the target video may be marked, edited, or timestamped to indicate the time position of the clips. In the embodiment, the target video is divided into multiple parts, each corresponding to an identified clip.

[0105] Translated subtitles In embodiments, the technology can generate one or more translated subtitles for a target video using subtitles generated in the target video's original audio language (spoken language, original language). As already described, the caption text in the original language is either provided as a script or generated by one or more audio processing techniques (STT) or other AI methods. The caption text in the original language may be translated into a desired foreign language (the language to which the caption text is translated will hereafter be referred to as the “translated language” or “second language”).

[0106] Translation by sentence In this embodiment, the technology translates caption text sentence by sentence into the target language. If the caption text in the original language contains sentence-ending markers, sentences separated by the sentence-ending markers may be translated individually. If the caption text in the original language does not contain punctuation information, as in the case of STT text, the technology performs sentence by sentence using sentences identified using a sentence model. Figure 14 shows translation 1400 of Korean STT text 920. Korean K1400 includes three translated sentences 1410, 1420, and 1450, which correspond to the original language texts 1110, 1120, and 1150, respectively.

[0107] In certain embodiments, two or more sentences may be translated together. Sentence-by-sentence translation may use a sentence (an identified sentence) as the unit of translation, but it may allow two or more sentences to be translated together. In embodiments, translation units other than sentences may be used (e.g., word by word, phrase by phrase, clip by clip, or a combination of different translation units).

[0108] Translated subtitles - clips based on the original language caption text In embodiments, to provide translated subtitles, the translated caption text in the target language can be synchronized with the audio of the target video in the original language by using or employing the same clip ("original clip") defined based on the caption text in the original language (defined by the timestamp in the original language). As described above, the technique may determine the clip based on sentence-ending and sentence-ending delimiters identified using the sentence model and sentence-ending model in the original language. The technique may use this to determine the clip for translated subtitles as well as for subtitles in the original language.

[0109] However, in some embodiments, the technique may process the translated caption text date to identify the end of the sentence and identified in the middle of the sentence using the translated language version of the sentence and an in-sentence model, and may determine a clip that is different from a defined clip based on the subtitle text in the original language.

[0110] Assigning translated caption text to the original clip. In the embodiment, if the translated subtitle follows the original clip and the clip contains only sentences, the entire translated sentence may be assigned to the same clip. If the clip contains two or more sentences, the entire translated sentence may be assigned to the same clip, maintaining the same order of sentences.

[0111] In the embodiment, when the translated subtitle follows the original clip(s), and the sentence is split into two or more clips (using the sentence and in-sentence models), the translated sentence can be split into the same number of clips so that the original sentence and the translated sentence are synchronized when the original subtitle and the translated subtitle are displayed together.

[0112] Split the translated text into the same number of clipped texts as the original sentences. In the embodiment, when a sentence in the original language is split into two or more clips in the subtitles of the original language, the translated text may be further processed because it may lack punctuation or other indications for splitting the original clip into two or more clips.

[0113] This technology can deploy a third AI model for identifying in-sentence segments in a translated text. The third AI model may be a version of the in-sentence model that is trained and configured in the translation language ("translated in-sentence model"). The translated in-sentence model may be deployed for each translated text and then aims to divide the text into a number of sentence parts that match the number of clips the text is divided into in the original language.

[0114] According to Figures 15A to 16, the in-sentence model 1510 (for example, the Korean version of model 1210) calculates the in-sentence segmentation probability 1520 (in percentage) for words in the translated sentence 1410 of the original language sentence 1110. Model 1510 did not calculate the probability for word 1524 because it is the sentence-ending word. In the embodiment, when the original language sentence 1110 is divided into two clip texts 1240 and 1260 to form the original language subtitle data 1300, the translated sentence 1410 is divided into the same number (2) of clip texts 1610 and 1620. In order to divide the sentence into "n" (a natural number greater than 2) of clip texts, "n-1" words (or more) with the highest in-sentence segmentation probability are determined as the in-sentence clip-ending words (or more). In Figure 15A, word 1522, which has the highest in-sentence segmentation probability (84%), is identified as the only clipping-ending word in the translated text 1410. In Figure 15B, the translated text 1410 is split into two clipping texts 1542 and 1544 to obtain the set of translated clipping texts 1530.

[0115] This selection can be made regardless of whether the word's in-sentence segmentation probability (84%) is greater than a predetermined threshold (e.g., 90%) for identifying the clipping end word in the original language sentence. In embodiments, even if there are two or more words with in-sentence segmentation probabilities greater than the predetermined threshold, only the single word ("n-1") with the highest in-sentence segmentation probability can be identified as the clipping end word, and the translated text can be split into two ("n") clipped texts.

[0116] Generation of translated subtitle data with the same time codes as the original language subtitles. In one embodiment, referring to Figure 16, the translated subtitle data (Korean) 1600 can be obtained by replacing the text in the original language subtitle 1300 not word by word, but clip by clip. Each clip text 1240, 1260, and 1120 of the original language subtitle 1300 is replaced with its corresponding translated text clips 1542, 1544, and 1420, respectively.

[0117] In this embodiment, the sequence number and timecode of the original language subtitle 1300 can be maintained within the translated subtitle data 1600. In the translated subtitle data 1600, the first translated clip text 1542 replaces the first original language clip text 1240, while maintaining the same sequence number 1312 and timecode 1314. Since the translated clip text 1542 is not spoken in the video, the timing of displaying the translated clip text 1542 is determined so that the translated clip text 1542 is synchronized with the sound of the corresponding clip text 1240.

[0118] Input / output of the model within the translated text In the embodiment, the translated in-sentence model is very similar to the source language in-sentence model described above, trained in a specific language, specialized for that language, and divides sentences in that language into sentence parts or segments. The construction, training, and operation of the translated in-sentence model can be understood by referring to the construction, training, and operation of the source language in-sentence model.

[0119] In one embodiment, the in-translation model is provided with individual sentences as input, and in this embodiment, each sentence has 30 words or less in the translation language. Next, in one embodiment, the in-translation model can identify sentence segments in the translation by identifying the word that has the highest probability of being the word immediately preceding a comma, i.e., its comma probability; in another embodiment, it can identify sentence segments in the translation by identifying the word that has the punctuation probability of a punctuation mark that functions as a sentence segment in the translation language. As a result, words that satisfy or exceed a threshold probability may be identified as sentence segment words in the translation language.

[0120] Sentence segments as output of the in-translation model In embodiments, the technology may then attempt to match a clip defined in the first language to a sentence and / or part of a sentence in the translation language if the clip defined in the first language exactly matches a sentence in the second language. If a complete sentence in one language is equivalent to a complete sentence in the second language, a perfectly matching clip is produced. Text from a text file in the translation language is then provided to the clip, and the subtitle of the entire sentence may become the translated clip caption associated with the video and audio.

[0121] However, in the embodiment, if there are multiple sentence parts corresponding to each clip, the matching translated part output by the in-translation model may match the clip defined by the AI ​​model of the original language. The translated part is displayed as a translated clip caption, which may be displayed together with the clip caption of the original language.

[0122] No determination of sentence end in the translated language. In some embodiments, when the technology performs sentence-by-sentence translation, individual sentences in the source language are translated into individual sentences in the target language, so there may be no sentence-end determination or sentence definition using a target language sentence model. Subtitle generating software or a platform may have a target language sentence model for generating subtitles for videos recorded in the target language, but the target language sentence model may not be used in the process of generating subtitles by translating subtitles in the source language.

[0123] Live video or recorded video This technology can be implemented, carried out, or executed on stored or pre-saved video. In embodiments, this technology can be applied to provide subtitles to a live video stream and to generate subtitles in real time while the video is being recorded live.

[0124] Processing timing In the embodiment, the process or action for providing subtitles to the video may be performed when the video is being recorded, when the video recording is paused, stopped, or terminated, or when the video is stored or loaded into a specific application, computing device, storage device, or cloud network.

[0125] Figure 1 Figure 1 is a flowchart illustrating an exemplary method 100 for providing subtitles to a video. The video is received or loaded from a local or remote data store 105 for further processing. At least one text file associated with the video or its associated audio (e.g., dubbed audio) is received or retrieved (110). The text file is input into a sentence model (115). The sentence model identifies one or more sentence-ending words based on sentence-ending probability values. Words that meet or exceed a predetermined threshold probability value may be identified as sentence-ending words. The location of sentence-ending words may be marked and / or timestamped in the text file and / or video file. The identified sentences are defined and output (120). The sentences are then input into an intrasentence model (125). The intrasentence model is run individually for each sentence and attempts to define one or more sentence segments or parts by identifying intrasentence boundaries. Sentence boundaries can be detected if a word has a sentence boundary probability that meets or exceeds a certain probability threshold, which the sentence model may consider to be immediately preceding a comma or another sentence boundary punctuation mark that functions as a sentence boundary. Sentence portions separated by the identified sentence boundaries are obtained (130). Based on the identified sentence ends and identified sentence boundaries, clips (clip text and corresponding clip durations) are defined (135), and subtitle files are generated according to the defined clips. When the video is played, the clip text is displayed sequentially as subtitles during the corresponding clip durations (140).

[0126] Figure 2 Figure 2 is a flowchart illustrating an exemplary method 200 for generating and displaying translated subtitles for a video. The video is received or loaded for further processing (205). A text file (in the original spoken language) associated with the video or its associated audio is received (210). Steps 215, 220, 225, 230, and 235 for defining clips are the same as steps 115-135 in Figure 1. Next, sentences identified from the text file are translated individually (240) to obtain the translated sentence. Next, an in-translation model is run on the translated sentence (245) to determine the in-sentence boundaries of the translated sentence. The clips of the translated sentence are defined to match the clips already defined in step 235 for the text file in the original spoken language. For example, if a sentence in the original language is split into two clips, the in-translation model aims to generate two sentence parts from the translated sentence to define the same number of clips as the sentence in the original language. Translated subtitles may be generated based on clips defined for the translation such that each clip text in the translated subtitle and the corresponding clip text in the original language subtitle share the same (or substantially the same) clip duration (250). When the video is played, the clip texts of the translated subtitle appear sequentially as subtitles for each clip duration (255).

[0127] Figure 3 Figure 3 presents a platform user interface in which subtitles are combined with video clips and the ability to edit video clips and clip captions simultaneously is provided. The user interface ("UI") 300 may be deployed as part of video editing software or an application and may include a video playback screen 305 in which a selected video clip or a complete video can be played along with subtitles linked to the video. The UI 300 may also include a clip editor panel 310 for one or more selected clips, which presents editable manually entered captions in a field 315. The clip editor panel 310 may also include video cuts 320, 325 for each selected clip. Each video cut is a segment of a video clip and is identified by a subtitle / caption word, punctuation, sound, or pause connected to that part of the clip. This allows editing of each position or segment of the clip by applying edits to the displayed word, punctuation, sound, or pause, thereby instantly applying the same edits to the corresponding segments within the clip. This makes video editing, including pausing, muting, or removing specific undesirable sections, much more efficient.

[0128] Therefore, each identified word, punctuation, or pause representing a video cut can be individually selected, edited, deleted, and manipulated, which directly affects the portion of the clip corresponding to the word representing the video cut. For example, if a clip consists of a speaker saying the phrase "I have a dream," deleting the video cut identified by the word "I" will automatically delete its associated linked video and audio as well, leaving only the subtitle / caption text, video, and audio for "have a dream" in the clip. After the deletion is made, playing the video will only play the "have a dream" portion.

[0129] If there are silences or pauses identified and / or marked by punctuation or pauses in the linked text file, these may also be displayed as individual video cuts 320, and their removal and deletion would allow for easy and automatic removal of the corresponding video and audio portions (i.e., pauses or silences) within the clip. UI300 may also include buttons to instantly remove all identified pauses, stops, silences, selected words or punctuation, and / or other unwanted sounds from one or more clips, or from the entire video, with a single click. Thus, UI300 makes it much easier to edit and delete unwanted portions or video cuts 320 from video clips through the link between the subtitle text and the corresponding video / audio portions.

[0130] Figure 4 Figure 4 shows a graphical representation of video 410 and its corresponding script text 415 (STT text) displayed along a timeline. The script text 415 is presented as blocks of words, each block of words representing the duration of the corresponding word in the video. The audio of word 1 is in the video. 10 It starts with, t 11 It ends there. In the embodiment, the script text 415 is obtained from the STT processing of the video audio, and word 1 is the video audio t 10 from t 11 It is obtained by transcribing to. In this embodiment, the timestamp (timecode) of word 1 is t 10 and t 11 It can be either one or both and can be included in the video's subtitle data. Word 1 and Word 2 are t 11 from t 21It is separated by silence until. In an embodiment, the sentence model processes the script text 415 to calculate the end-of-sentence probability (or start-of-sentence probability) of one or more words in the text. Word 3 is determined to be the ending word of sentence 1 based on its end-of-sentence probability. In an embodiment, the sentence model identifies or determines a punctuation mark that ends a sentence in the text 415. For example, in FIG. 4, the sentence model determines a period as the end-of-sentence punctuation mark for sentence 1 and a question mark for sentence 3.

[0131] Figure 5 FIG. 5 shows a graphical representation of a clip 510 defined for the video 410 and the script text 415. In an embodiment, the in-sentence model processes the sentences in the text 415 to detect one or more in-sentence delimiters. For example, in FIG. 4, the in-sentence model determines that a comma is used as an in-sentence delimiter in sentence 2. Based on the end-of-sentence punctuation mark and the in-sentence delimiter, sentence 1 forms the clip text of the sling clip 511. The sling clip 511 is defined by its start time t 10 (start time code of the first word 1) and the end time t 32 (end time code of the last word 3). The time codes t 10 and t 32 are included in the subtitle data of the video to indicate the timing of presenting words 1 to 3. In an embodiment, the word time code t 21 is not excluded from the subtitle data if it does not affect presenting words 1 to 3 as subtitles. The time code t 52 (end of the word 5 immediately before the identified comma) and the time code t 61 (start of the word 6 immediately after the identified comma) are used to split sentence 2 into two clips 512 and 513, and words 4 to 5 form the clip text of clip 512, and words 6 to 9 form the clip text of clip 513.

[0132] Figure 6 Figure 6 shows a graphical representation of the association between the original spoken language script text 415 and the translated script text 610 in order to provide subtitle data in the translated language. In the embodiment, the translated script text 610 is obtained by translating the original language text 415 sentence by sentence. The translation of sentence 1 itself forms the translated clip text of clip 512. The translation of sentence 2 is divided into the same number of clip texts (2) as sentence 2 in the original language, regardless of the number of intra-sentence segments identified by running an intra-sentence model (in the translated language) on the translated sentence 2 (for example, even if it would be more natural without intra-sentence segments, translated sentence 2 is divided into 2 to match the number of clip texts identified in sentence 2 in the original language). Clips 511-513, defined based on the original language text 415, are maintained when creating the translated subtitle data because the translated text 610 does not have its own timecode. In the embodiment, words may be swapped or placed in different clips, even if they directly match in the translated text to make sense. For example, word 4 is translated to word tr5 in the translation language, but word tr5 is displayed as a subtitle in clip 513 and word 4 is displayed as a subtitle in clip 512. In this embodiment, the clip-ending words in the original language and the translated language do not have the same meaning. For example, word 3 and word tr3, which end in the same clip 511, do not have the same meaning because the translation of word 3 is word tr2 and not word tr3.

[0133] Figures 7A and 7B Figures 7A and 7B illustrate embodiments of a method for creating subtitles in one or more languages ​​for a video clip by detecting sentence endings, sentence breaks, and translations into an optional other language. In this embodiment of the method for creating subtitles from video 700, the video is received by a system or platform (705) and may be received along with one or more associated text files (710). These text files may contain transcripts or pre-made subtitles associated with the video. One or more text files may be additionally or alternatively generated by the Technique via the AI ​​models or audio extraction or transcription methods described herein (715). The sentence model may be run on the text files, taking in text of a specific maximum input size for efficient execution. If the probability value of a word meets or exceeds a threshold (which may be predefined or determined by the sentence model or otherwise), the word is marked or identified as a sentence-ending word (725), and the location of the identified sentence-ending word is marked in the text file and / or the corresponding video and associated audio files (730). Marking can be done in any appropriate way, such as by adding timestamps, manipulating or modifying metadata or any related text, audio or video files, or any related video editing files. After the position of the sentence-ending word is established, the video is divided into several clips (735). Each clip contains only one complete sentence. Each clip ends with the sentence-ending word. Next, a second AI model, the sentence model, can be run in a similar manner on the sentences / clips generated by the sentence model to further refine the clips generated by identifying sentence-ending divisions (740).

[0134] Sentence breaks are identified by the probability of a word being preceded by a comma or punctuation mark in a text file, i.e., the probability that a word precedes a punctuation mark that could divide a sentence. In embodiments, a word may have different punctuation probabilities for each adjacent space, i.e., different punctuation probabilities for both sides of a word. For example, a particular punctuation mark may have a 50% probability of being immediately to the left of a word and an 85% probability of being to the right. One side of a word may also include spaces between the punctuation mark, either before or after the word and the space. In embodiments, the punctuation probability for only one side of a word may be considered, which may vary depending on the language of the text. For example, in English, a space immediately to the right of a word is the space following the word and is generally considered. If the punctuation probability of a word meets or exceeds a threshold (which may be predefined or determined by an AI model or any other method), the word is identified as adjacent to punctuation (745), or a particular space is identified as sentence punctuation or a comma (745).

[0135] Once an intra-sentence segment is identified (745), its location may be marked within the text file and / or the corresponding video location (750). Marking may be done in any appropriate way, such as by adding a timestamp, manipulating or modifying metadata or any relevant data within the text, an audio or video file, or any relevant video editing file. Optional or additional AI refinement models may be deployed to identified sentences and / or sentence segments within the text file (755) and may use other factors to identify sentence ends and intra-sentence segments. These factors may include, but are not limited to, pauses or silences within a clip or audio, specific phrases, specific words, or one or more of the length of a clip, the length of a sentence, or other user configurations and settings. Optionally, the original text file may be translated into another language (760). Translation into the target language (760) is performed sentence by sentence using the sentences identified by the sentence model. The translated sentences are then input into the intra-sentence translation AI (765) so that intra-sentence segments matching defined clips can be identified. Next, the translation model outputs translated captions that match the structure of the defined clip (770), the positions of these sentences / sentence portions may then be marked in the translated text file and / or video file (775), and these are matched with the corresponding video clips (780). The translated captions are displayed along with the original language subtitles of each video clip (785). Subsequently, some or all of the generated captions in any or all languages ​​may be displayed on the user interface (790), the clip captions are combined with the corresponding video clips and can be edited together on the user interface. For an example of the user interface, see Figure 3 of this application.

[0136] Figure 8 - Example architecture of a user computing system Figure 8 shows an architecture of an exemplary computing device 800 that can be used to perform one or more features of the present technology. The overall architecture of computing device 800 includes an arrangement of computer hardware modules and software modules that can be used to implement one or more aspects of the present disclosure. Computing device 800 may include more (or fewer) elements than those shown in Figure 8. However, it is not necessary to show all of these elements in order to provide a valid disclosure.

[0137] The illustrated computing device 800 includes a processor 810, a network interface 820, a computer-readable medium 830, and an input / output device interface 840, all of which can communicate with each other via a communication bus. The network interface 820 may provide connectivity to one or more network or computing systems. The processor 810 may also communicate with memory 850 and, via the input / output device interface 840, may further provide output information to one or more output devices such as a display (e.g., display 841) and a speaker. The input / output device interface 840 may also accept input from one or more input devices such as a camera 842 (e.g., a 3D depth camera), a keyboard, a mouse, a digital pen, a microphone, a touchscreen, a gesture recognition system, a speech recognition system, an accelerometer, and a gyroscope.

[0138] The memory 850 may contain computer program instructions (grouped as modules in some implementations) executed by the processor 810 to implement one or more aspects of the present disclosure. The memory 850 may include RAM, ROM, and / or other persistent, auxiliary, non-temporary computer-readable media.

[0139] The memory 850 may store an operating system 851 that provides computer program instructions for use by the processor 810 in the general management and operation of the computing device 800. The memory 850 may further include computer program instructions and other information for implementing one or more aspects of the present disclosure.

[0140] In one embodiment, for example, memory 850 includes a user interface module 852 that generates a user interface (and / or instructions for it) to be displayed via a browser or application installed on a computing device 800, for example. In addition to the user interface module 852, and / or in combination with the user interface module 852, memory 850 may include a video processing module 853, a text processing module 854, and a machine training model 854, which can be executed by the processor 810.

[0141] In the example shown in Figure 8, a single processor, a single network interface, a single computer-readable medium, a single input / output device interface, a single memory, a single camera, and a single display are shown. However, in other implementations, the computing device 1500 may have one or more of these components (e.g., two or more processors and / or two or more memories).

[0142] Processing using remote computing devices In embodiments, one or more processes of the Technology may be executed by an exemplary computing device 800, by a remote server, or by a combination of the exemplary computing device 800 and a remote server. For example, if a smartphone does not have a machine-trained model in its local data store, it can communicate with a remote computing server or a cloud computing system to execute one or more processes of the Technology.

[0143] Computer executable instructions A logical block, module, or unit described in connection with an implementation disclosed herein can be implemented or executed by a computing device having at least one processor, at least one memory, and at least one communication interface. Elements of a method, process, or algorithm described in connection with an implementation disclosed herein can be implemented directly in hardware, as a software module executed by at least one processor, or a combination of both. Computer-executable instructions for implementing a method, process, or algorithm described in connection with an implementation disclosed herein can be stored in a non-temporary computer-readable storage medium.

[0144] Alternative implementations and obvious corrections While implementations of the present invention have been disclosed in relation to specific implementations and embodiments, it will be understood by those skilled in the art that the present invention extends beyond the specifically disclosed implementations to other alternative implementations and / or uses of the present invention, as well as obvious modifications and equivalents thereof. In addition, while numerous variations of the present invention have been shown and described in detail, other modifications within the scope of the present invention will be readily apparent to those skilled in the art based on this disclosure. It is also intended that various combinations or partial combinations of specific features and aspects of the implementation may be made within the scope of one or more of the present invention. Accordingly, it should be understood that the various features and aspects of the disclosed implementations may be combined with or substituted for each other to form various forms of the disclosed invention. Accordingly, the scope of the present invention disclosed herein should not be limited by the specific disclosed implementations described above, and various modifications in form and detail may be made without departing from the spirit and scope of this disclosure as set forth in the claims.

Claims

1. A computer implementation method for providing subtitles to a video, which is executed by at least one processor that executes instructions stored in a non-temporary computer-readable medium, and includes the following: Processing the audio data of the video to generate a timed script in a first language that includes a first word sequence and a timestamp for each word in the first word sequence, Processing the first word sequence and using a first machine-trained model to calculate the sentence-end probability of at least one word in the first word sequence, Based on the sentence-end probability of the first word, the first word in the first sequence is determined to be the first sentence-end word, and a first sentence ending with the first word is defined. Processing the first sentence and using a second machine-trained model to calculate the in-sentence segmentation probability of at least one word in the first sentence, Determining the second word of the first sentence as a clip-ending word based on the in-sentence segmentation probability of the second word, defining a first clip text ending with the second word, the definition of the first clip text further defining a first clip duration corresponding to the first clip text and ending when the second word is spoken in the video, This includes generating subtitle data in a first language, which includes the first clip text and information indicating the first clip period in which the first clip text is displayed as subtitles in a first language. The above method further, The first sentence is translated into a first translated sentence in a second language, wherein the first translated sentence ends with the first translated word. The first translated sentence is processed, and the third machine-trained model is used to calculate the in-sentence segmentation probability of at least one word in the first translated sentence. Based on the in-sentence segmentation probability of the second translated word, the second translated word of the first translated sentence is determined as the clipping end word, and the first translated clipping text ending with the second translated word is defined as the determination, The method includes generating subtitle data in a second language, which includes the first translated clip text and information indicating the second language period in which the first translated clip text is displayed as subtitles in the second language. The method wherein the first clipping period for displaying the first clipping text is the same as or substantially the same as the second language period for displaying the first translated clipping text, regardless of whether the second word ending the first clipping text corresponds semantically to the second translated word ending the first translated clipping text.

2. The timed script includes a timestamp of the second word indicating the time the second word was spoken in the video. The method according to claim 1, wherein generating subtitle data in the second language includes specifying a timestamp of the second word as the end of the first clip period and the second language period in the subtitle data in the second language according to a predetermined subtitle format.

3. The first clip text begins with the third word of the first sentence, and the first translated clip text begins with the third translated word of the first translated sentence. The timing script includes a timestamp of the third word indicating the time when the sound of the third word begins in the video. The method according to claim 2, further comprising generating subtitle data in the second language by specifying in the subtitle data in the second language the timestamp of the third word as the start of the first clip period and the second language period, such that the first clip period and the second language period are the same regardless of whether the third word semantically corresponds to the third translated word.

4. The first translation does not include punctuation marks indicating internal divisions within the first translation. The method according to any one of claims 1 to 3, wherein the third machine-trained model is trained using a plurality of punctuated sentences in the second language and is configured to calculate the probability that at least one intrasentence punctuation mark follows at least one word in the input sentence.

5. The first sentence is divided into n clipped texts based on at least the first word and the second word, where n is a natural number greater than 2. The method according to any one of claims 1 to 3, wherein the first translated text is divided into the same "n" translated clip texts.

6. The process further includes determining the fourth word of the first sentence as the clipping end word, defining a second clipping text that begins with the word immediately following the second word and ends with the fourth word, The definition of the second clip text further defines a second clip that corresponds to the second clip text and ends when the fourth word is spoken in the video, The method according to claim 5, wherein the fourth word in the first sentence is identified as a clip-ending word based on at least one of the in-sentence segmentation probability of the fourth word and the length of silence following the sound of the fourth word in the video.

7. The subtitle data in the first language is configured such that the entire first clip text is displayed as a subtitle for the video at the start of the first clip period and is maintained without interruption until the end of the first clip period. The method according to any one of claims 1 to 3, wherein the subtitle data for the second language is configured such that the entire first translated clip text is displayed as a subtitle for the video at the start of the second language period and maintained without interruption until the end of the second language period.

8. The method according to any one of claims 1 to 3, wherein generating subtitle data in the second language includes specifying the time when the second word was spoken in the first language as the end of the second language period, such that the first clip period and the second language period end simultaneously.

9. The method according to any one of claims 1 to 3, wherein the first translated clip text in the second language has no meaning corresponding to the first clip text in the first language.

10. The method according to any one of claims 1 to 3, wherein the first translated clip text has meaning corresponding to the first clip text in the first language.

11. Processing the audio data of the video to generate the timed script includes performing speech-to-text (STT) processing on the audio data, in which the audio corresponding to the second word is transcribed to the second word, the time at which the second word was spoken in the video is determined, and the time at which the second word was spoken is specified in the timed script for the second word. The information indicating the first clipping period includes the time during which the second word, determined by the STT processing, was spoken. The method according to any one of claims 1 to 3, wherein generating subtitle data for the first language includes associating the time at which the second word was spoken, as determined by the STT processing, with the first clip text as the end of the first clip period, in accordance with a predetermined subtitle file format.

12. Processing the audio data of the aforementioned video to generate the timingd script is, The identification of silence and non-silence within the audio data, wherein the non-silence includes the corresponding sound of the second word, The first word sequence is obtained by transcribing the corresponding sounds of the second word to the second word, With respect to the second word, the end time at which the corresponding sound of the second word ends in the video is determined, The method according to any one of claims 1 to 3, further comprising including the determined end time as a timestamp of the second word in the timed script.

13. Processing the audio data of the aforementioned video to generate the timingd script is, Obtaining a prewritten script for the video that includes the first word sequence but does not include a timestamp for the first word sequence, For each word in the first word sequence, identify the corresponding sound in the audio data that identifies the first sound corresponding to the second word, The end time of the first sound is determined when the first sound ends in the video, The method according to any one of claims 1 to 3, comprising combining the determined end time and the first word sequence to generate the timed script such that the determined end time is specified as the timestamp of the second word in the timed script.

14. A computer implementation method for providing subtitles to a video, which is executed by at least one processor that executes instructions stored in a non-temporary computer-readable medium, and includes the following: Processing the audio data of a video to generate a timed script in a first language, which includes a first word sequence and a timestamp for each word in the first word sequence. Processing the first word sequence and using a first machine-trained model to calculate the sentence-end probability of at least one word in the first word sequence, Based on the sentence-end probability of the first word, the first word in the first sequence is determined to be the first sentence-end word, and a first sentence ending with the first word is defined. Processing the first sentence and using a second machine-trained model to calculate the in-sentence segmentation probability of at least one word in the first sentence, Determining the second word of the first sentence as a clip-ending word based on the in-sentence segmentation probability of the second word, defining a first clip text ending with the second word, the definition of the first clip text further defining a first clip period corresponding to the first clip text and ending when the second word is spoken in the video, and The method comprising generating subtitle data in a first language, which includes the first clip text and information indicating the first clip period in which the first clip text is displayed as subtitles in a first language.

15. The timed script includes a timestamp of the second word indicating the time the second word was spoken in the video. The method according to claim 14, wherein generating subtitle data in the first language includes specifying a timestamp of the second word as the end of the first clipping period according to a predetermined subtitle format.

16. The first clip text begins with the third word of the first sentence, The timing script includes a timestamp of the third word indicating the time when the sound of the third word begins in the video. The method according to claim 15, wherein generating subtitle data in the first language includes specifying the timestamp of the third word as the start of the first clipping period in accordance with the predetermined subtitle format.

17. The method according to claim 16, wherein the subtitle data in the first language is configured such that the entire first clip text is displayed as a subtitle for the video at the start of the first clip period and is maintained without interruption until the end of the first clip period.

18. The method according to claim 16, wherein the first clip text further includes a fourth word between the third word and the second word, and the subtitle data in the first language does not include a timestamp of the fourth word, so that the first clip text is displayed as a subtitle without referring to the fourth word.

19. The aforementioned timing script does not include punctuation marks indicating the end of the first sentence, or internal sentence breaks within the first sentence. The first machine-trained model is trained using multiple punctuated texts, each containing one or more sentence-ending punctuation marks, and is configured to calculate the probability that at least one sentence-ending punctuation mark follows immediately after at least one word in the input text. The method according to any one of claims 14 to 18, wherein the second machine-trained model is trained using a plurality of punctuated texts, each containing one or more intermelligent punctuation marks, and is configured to calculate the probability that at least one intermelligent punctuation mark follows at least one word in an input sentence.

20. The method according to any one of claims 14 to 18, further comprising determining the fifth word of the first sentence as another clip-ending word based on the in-sentence segmentation probability of the fifth word, thereby defining a second clip text that begins with the word immediately following the second word and ends with the fifth word, the definition of the second clip text further defines a second clip corresponding to the second clip text and ending when the fifth word is spoken in the video.

21. Processing the audio data of the video to generate the timed script includes performing speech-to-text (STT) processing on the audio data, in which the audio corresponding to the second word is transcribed to the second word, the time at which the second word was spoken in the video is determined, and the time at which the second word was spoken is specified in the timed script for the second word. The information indicating the first clipping period includes the time during which the second word, determined by the STT processing, was spoken. The method according to any one of claims 14 to 18, wherein generating subtitle data for the first language includes associating the time at which the second word was spoken, as determined by the STT processing, with the first clip text as the end of the first clip period, in accordance with a predetermined subtitle file format.

22. Processing the audio data of the aforementioned video to generate the timingd script is, The identification of silence and non-silence within the audio data, wherein the non-silence includes the corresponding sound of the second word, The first word sequence is obtained by transcribing the corresponding sounds of the second word to the second word, With respect to the second word, the end time at which the corresponding sound of the second word ends in the video is determined, The method according to any one of claims 14 to 18, further comprising including the determined end time as a timestamp of the second word in the timed script.

23. Processing the audio data of the aforementioned video to generate the timingd script is, Obtaining a prewritten script of the video that includes the first word sequence but does not include a timestamp for the first word sequence, For each word in the first word sequence, identify the corresponding sound in the audio data that identifies the first sound corresponding to the second word, The end time of the first sound is determined when the first sound ends in the video, The method according to any one of claims 14 to 18, comprising combining the determined end time and the first word sequence to generate the timed script such that the determined end time is specified as the timestamp of the second word.

24. The method according to any one of claims 14 to 18, wherein the information indicating the first clip period includes a first timestamp indicating the start time of the first clip in the video, and further includes a second timestamp indicating the end time of the first clip in the video, and the first clip text is displayed together with the video without interruption from the start time of the first clip to the end time of the first clip.

25. The method according to any one of claims 14 to 18, wherein the timestamp for each word in the first word sequence defines the time at which the sound of the corresponding word ends or begins in the video.

26. The method according to any one of claims 14 to 18, wherein the timed script further comprises a second word sequence, and the timed script further comprises a timestamp for at least one word in the second word sequence.

27. A non-temporary computer-readable medium that, when executed by at least one processor, stores instructions causing the at least one processor to perform the method according to claim 1.

28. A system that provides subtitles for videos, At least one processor, The system comprises at least one memory, wherein, when executed by the at least one processor, the memory stores instructions causing the at least one processor to perform the method according to claim 1.

29. A non-temporary computer-readable medium that, when executed by at least one processor, stores instructions causing the at least one processor to perform the method according to claim 14.

30. A system that provides subtitles for videos, At least one processor, The system comprises at least one memory, wherein, when executed by the at least one processor, the memory stores instructions causing the at least one processor to perform the method according to claim 14.

Images