A subtitle processing method, system, electronic device and storage medium

Abstract

The application provides a subtitle processing method and system, electronic equipment and a storage medium, which are applied to the technical field of data processing, and multiple sentences and original subtitle sentence time lengths thereof are generated according to a subtitle file; wherein a sentence is composed of at least one piece of subtitle, and the original subtitle sentence time length is the sum of the original subtitle time lengths of all subtitles constituting the sentence; the sentence is translated to obtain a translation of the sentence; based on the sentence and the original subtitle sentence time length thereof, the translation of the sentence is split with minimum deviation to obtain a splitting scheme of the sentence; wherein the splitting scheme comprises a translation fragment of each piece of subtitle corresponding to the sentence; the translation fragment of each piece of subtitle is dubbed, and the dubbing speed of the translation fragment dubbing of each piece of subtitle is adjusted based on the original subtitle time length of the subtitle, so that the purpose of high-precision time alignment is achieved under the condition that the sentence is not fragmented, the semantics is coherent, the semantics is complete, and the watching process is ensured.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a subtitle processing method, system, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the film and television industry, film and television content is also being disseminated globally. In order to improve people's viewing experience, subtitles can be translated into film and television content.

[0003] Current subtitle translation and dubbing technologies primarily rely on manual processing or a combination of basic machine translation and human proofreading. However, both methods suffer from inconsistencies in terminology translation, semantic breaks, and low translation efficiency. Furthermore, the dubbing process is prone to producing unnatural voiceovers and high rework rates. Summary of the Invention

[0004] In view of this, the present invention provides a subtitle processing method, system, electronic device and storage medium to solve the problems of inconsistent terminology translation, semantic breaks, low translation efficiency, unnatural dubbing and high rework rate in the prior art.

[0005] The first aspect of this application provides a subtitle processing method, the method comprising:

[0006] Multiple sentences and their original subtitle durations are generated based on the subtitle file; wherein, each sentence consists of at least one subtitle, and the original subtitle duration is the sum of the original subtitle durations of all subtitles that constitute the sentence;

[0007] The sentence is translated to obtain its translated form;

[0008] Based on the sentence and the duration of its original subtitle, the translation of the sentence is segmented with minimum deviation to obtain a segmentation scheme for the sentence; wherein, the segmentation scheme includes the translated segments of each subtitle corresponding to the sentence;

[0009] The translated text segment of each subtitle is dubbed, and the dubbing rate of the dubbing of the translated text segment of each subtitle is adjusted based on the original subtitle duration.

[0010] Optionally, multiple sentences and their original subtitle durations can be generated based on the subtitle file, including:

[0011] Obtain a subtitle file, wherein the subtitle file includes multiple subtitles;

[0012] The subtitle file is traversed using a preset sliding window, and the sentence end positions in the subtitle text within the preset sliding window are identified by a large language model until the entire subtitle file is traversed, resulting in multiple sentence end positions.

[0013] The subtitles are merged according to the end position of each sentence to obtain multiple sentences;

[0014] Generate a set of original subtitle indexes for the sentence, and generate the original subtitle duration for the sentence based on the set of original subtitle indexes.

[0015] Optionally, the sentence is translated to obtain a translated version, including:

[0016] The translation model is used to translate each subtitle of the sentence, resulting in multiple translation results;

[0017] If the number of multiple translation results is not equal to the number of subtitles in the sentence, the first subtitle that has not been translated is split, and the split result of the first subtitle is translated using the translation model to obtain the translation result of the first subtitle;

[0018] If the sum of the number of multiple translation results and the number of translation results of the first subtitle is not equal to the number of subtitles, the sentence-by-sentence translation mode is used to translate the second subtitle that has not been translated in the sentence, and the translation result of the second subtitle is obtained;

[0019] The translation of the sentence is generated based on the multiple translation results, the translation result of the first subtitle, and the translation result of the second subtitle.

[0020] Optionally, based on the duration of the sentence and its original subtitle, the translation of the sentence is segmented using minimum deviation to obtain a segmentation scheme for the sentence, including:

[0021] The translated version of the sentence is dubbed to obtain the dubbed translation of the sentence;

[0022] Generate the translated word sequence and its timestamp sequence for the dubbing of the sentence, and perform minimum deviation segmentation of the translation of the sentence based on the original subtitle sentence duration, translated word sequence and timestamp sequence to obtain the segmentation scheme of the sentence.

[0023] Optionally, the translated text of the sentence is dubbed to obtain a dubbed translation of the sentence, including:

[0024] Obtain the term set of the sentence; wherein the term set includes multiple terms contained within the sentence;

[0025] The translation of the sentence is processed by speech synthesis to obtain the initial translation dubbing of the sentence and the duration of the initial translation dubbing;

[0026] Calculate the length ratio of the sentence based on the initial dubbing duration and the original subtitle duration;

[0027] If the duration ratio is within a preset range, the initial translated dubbing of the sentence and its initial translated dubbing duration will be used as the translated dubbing of the sentence and its translated dubbing duration.

[0028] If the duration ratio is not within a preset range, the translation of the sentence is optimized using the terminology set;

[0029] The optimized translation is used to update the translation of the sentence, and then the process of performing speech synthesis on the translation of the sentence is returned to obtain the initial translation dubbing of the sentence and its initial dubbing duration.

[0030] Optionally, a sequence of translated words and their timestamps for the dubbing of the sentence are generated, and the translation of the sentence is segmented with minimum deviation based on the original subtitle sentence length, the translated word sequence, and the timestamp sequence to obtain a segmentation scheme for the sentence, including:

[0031] The timestamp prediction model is used to predict the translated word sequence and timestamp sequence of the dubbing of the sentence.

[0032] A dynamic programming algorithm is used to perform minimum deviation segmentation of the translation of the sentence based on the original subtitle length, the translated word sequence, and the timestamp sequence, resulting in a segmentation scheme for the sentence.

[0033] A second aspect of this application provides a subtitle processing system, the system comprising:

[0034] A subtitle merging unit is used to generate multiple sentences and their original subtitle sentence durations based on a subtitle file; wherein, each sentence consists of at least one subtitle, and the original subtitle sentence duration is the sum of the original subtitle durations of all subtitles constituting the sentence;

[0035] A translation unit is used to translate the sentence to obtain a translated version of the sentence;

[0036] The translation optimization and segmentation unit is used to perform minimum deviation segmentation of the translation of the sentence based on the sentence and the duration of its original subtitle, to obtain a segmentation scheme for the sentence; wherein, the segmentation scheme includes the translated segments of each subtitle corresponding to the sentence;

[0037] The translation and dubbing synthesis unit is used to dub the translated text segment of each subtitle and adjust the dubbing rate of the translated text segment of each subtitle based on the original subtitle sentence length.

[0038] Optionally, a subtitle merging unit is generated based on the subtitle file, containing multiple sentences and their original subtitle durations. This is specifically used for:

[0039] Obtain a subtitle file, wherein the subtitle file includes multiple subtitles;

[0040] The subtitle file is traversed using a preset sliding window, and the sentence end positions in the subtitle text of the preset sliding window are identified by a large language model until the subtitle file is completely traversed, resulting in multiple sentence end positions.

[0041] The subtitles are merged according to the end position of each sentence to obtain multiple sentences;

[0042] Generate a set of original subtitle indexes for the sentence, and generate the original subtitle duration for the sentence based on the set of original subtitle indexes.

[0043] A third aspect of this application provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the subtitle processing method provided in the first aspect of this application.

[0044] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for performing the subtitle processing method provided in the first aspect of this application.

[0045] This application provides a subtitle processing method, system, electronic device, and storage medium. The method involves generating multiple sentences and their original subtitle durations based on a subtitle file; wherein each sentence consists of at least one subtitle, and the original subtitle duration is the sum of the original subtitle durations of all subtitles constituting the sentence; translating the sentences to obtain a translated text; performing minimum deviation segmentation on the translated text based on the sentences and their original subtitle durations to obtain a segmentation scheme for the sentences; wherein the segmentation scheme includes translated text segments for each subtitle corresponding to the sentence; dubbing each translated text segment of the subtitles, and adjusting the dubbing rate of each translated text segment of the subtitles based on the original subtitle duration. Therefore, this application first generates multiple sentences and their original subtitle durations based on the subtitle file; secondly, it translates the subtitles in the sentences to obtain complete translations of the sentences, so that the translations of the sentences can be segmented with minimal deviation. This allows the resulting sentence segmentation scheme to be directly applied on the timeline, which not only avoids sentence fragmentation and semantic incoherence, but also achieves high-precision time alignment while ensuring semantic integrity and viewing flow. This solves the problems of inconsistent terminology translation, semantic breaks, low translation efficiency, unnatural dubbing, and high rework rate in existing technologies. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 A flowchart illustrating a subtitle processing method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the structure of a subtitle processing system provided in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0052] As the background technology described above shows, current subtitle translation and dubbing technologies mainly rely on manual processing or a combination of basic machine translation and manual proofreading. However, both methods suffer from inconsistencies in terminology translation, semantic breaks, and low translation efficiency. Furthermore, the dubbing process is prone to producing unnatural voiceovers and high rework rates.

[0053] In existing technologies, subtitle translation and dubbing can also be achieved through end-to-end video translation solutions or dubbing duration matching methods. End-to-end video translation solutions can integrate technologies such as Automatic Speech Recognition (ASR), translation, speech synthesis, or Text-to-Speech (TTS). However, this approach focuses on process integration and pays insufficient attention to subtitle item alignment and precise segmentation, which can easily lead to inconsistencies in terminology translation and semantic breaks. Dubbing duration matching methods perform text normalization by calculating ratio coefficients for individual subtitles. This is suitable for scenarios where the subtitles themselves are complete, but in scenarios where segmentation is required, it can disrupt the integrity of sentences.

[0054] Therefore, it is evident that existing subtitle translation and dubbing methods suffer from inconsistencies in terminology translation, semantic breaks, and low translation efficiency. Furthermore, the dubbing process is prone to producing unnatural voiceovers and high rework rates.

[0055] This application provides a subtitle processing method, system, electronic device, and storage medium. First, multiple sentences and their original subtitle durations are generated based on the subtitle file. Second, the subtitles in the sentences are translated to obtain complete translations of the sentences. This allows for minimal deviation segmentation of the translated sentences, enabling the segmentation scheme to be directly applied on the timeline. This not only avoids sentence fragmentation and semantic incoherence but also achieves high-precision time alignment while ensuring semantic integrity and viewing flow. This solves the problems of inconsistent terminology translation, semantic breaks, low translation efficiency, unnatural dubbing, and high rework rates in existing technologies.

[0056] See Figure 1 The diagram illustrates a flowchart of a subtitle processing method provided in an embodiment of this application. The subtitle processing method specifically includes the following steps:

[0057] S101: Generate multiple sentences and their original subtitle durations based on the subtitle file; wherein, a sentence consists of at least one subtitle, and the original subtitle duration is the sum of the original subtitle durations of all subtitles that make up the sentence.

[0058] In the specific execution step S101, a subtitle file that needs to be translated and dubbed can be obtained in advance. The subtitle file includes multiple subtitles arranged in sequence. Multiple sentence end positions are identified from the subtitle file, and the subtitles are merged according to the sentence end positions to obtain multiple sentences. Each sentence consists of at least one subtitle.

[0059] It should be noted that the sentence can end at the position of a period, but this embodiment of the application does not limit this.

[0060] Optionally, the process of generating multiple sentences and their original subtitle durations based on the subtitle file can be as follows: obtaining a subtitle file, wherein the subtitle file includes multiple subtitles; traversing the subtitle file using a preset sliding window, and identifying the sentence end positions in the subtitle text within the preset sliding window using a large language model, until the subtitle file is completely traversed, resulting in multiple sentence end positions; merging the subtitles according to their respective sentence end positions to obtain multiple sentences; generating a set of original subtitle indexes for each sentence, and generating the original subtitle duration of the sentence based on the set of original subtitle indexes; wherein the original subtitle duration is the sum of the original subtitle durations of all subtitles constituting the sentence.

[0061] In some embodiments, the size of a preset sliding window (e.g., 30 subtitles) can be pre-configured. The preset sliding window is used to traverse multiple subtitles in the subtitle file from the beginning. The sentence end position is identified from the subtitle text in the current window using a large language model. The relative position in the current window is converted into a global position. The next window continues to traverse the subtitle file from the last segmentation point (the last sentence end position) of the current window until the subtitle file is traversed completely, so as to identify the sentence end positions in the subtitle file. The subtitles are merged according to the sentence end positions to obtain multiple complete sentences. The index of each subtitle constituting the sentence is obtained. The original subtitle index set of the sentence is constructed according to the index of each subtitle of the sentence, and the mapping relationship between the sentence and the original subtitle index set is constructed. The original subtitle sentence duration of the sentence is calculated according to the original subtitle duration of each subtitle of the sentence, as shown in formula (1).

[0062] (1)

[0063] Among them, I S Let T be the set of subtitle indices corresponding to sentence S, where i is the i-th subtitle, and T is the subtitle index set. sub,i The original subtitle duration is the length of the i-th subtitle. The duration of the original subtitle for sentence S.

[0064] In summary, this application uses a sliding window combined with a large language model to identify sentence boundaries (sentence end positions), merges subtitles according to sentence boundaries, and obtains the mapping relationship between sentence-original subtitle index set and the original subtitle duration of the sentence.

[0065] S102: Translate the sentence to obtain the translated version.

[0066] In the specific execution step S102, after obtaining multiple sentences from the subtitle file, for each sentence, each subtitle constituting the sentence can be translated to obtain multiple translation results, where each translation result is the translation of a subtitle; if multiple translation results meet the format verification conditions, the translation of the sentence is directly generated based on each translation result.

[0067] Optionally, the process of translating a sentence to obtain its translation can be as follows: Each subtitle in the sentence is translated using a translation model to obtain multiple translation results; if the number of multiple translation results is not equal to the number of subtitles in the sentence, the first subtitle that has not been translated is split, and the split results of the first subtitle are translated using a translation model to obtain the translation result of the first subtitle; if the sum of the number of multiple translation results and the number of translation results of the first subtitle is not equal to the number of subtitles, the second subtitle that has not been translated is translated using a sentence-by-sentence translation mode to obtain the translation result of the second subtitle; the sentence translation is generated based on the multiple translation results, the translation result of the first subtitle, and the translation result of the second subtitle.

[0068] In some embodiments, corresponding format validation conditions can be pre-configured, wherein the format validation conditions indicate that the sum of all translation results obtained during the translation of the sentence's subtitles is equal to the number of subtitles in the sentence. These "all translation results" can include multiple translation results, or the sum of multiple translation results and the translation results of each first subtitle, or the sum of multiple translation results, the translation results of the first subtitle, and the translation results of the second subtitle.

[0069] In practical applications, for each sentence in the subtitle file, a three-level degradation strategy of "large chunk → small chunk → sentence by sentence" can be adopted to complete the translation of the sentence. This ensures that the total number of all translation results obtained during the translation process is consistent with the number of subtitle entries in the sentence, so as to provide a high-quality translation with the same number of entries, and maintain the integrity of the sentence and the coherence of the context.

[0070] Specifically, each subtitle and its context in a sentence, along with a terminology dictionary, can be input into the translation model. The model then uses the context and terminology dictionary to perform a holistic translation of the input subtitles, resulting in a structured mapping of "index-translation." This structured mapping represents the translation of the subtitle corresponding to that index. The model then checks if the number of translation results matches the number of subtitles in the sentence. If they match, the sentence translation is considered complete, and the translation is generated based on the individual results. If they don't match, a downgrade check is performed: the first subtitle (considered a large subtitle block) that is not fully translated (no corresponding translation) is identified from the sentence's subtitles and broken down into smaller subtitles. The process involves re-translating the sentence; determining whether the total number of translation results obtained from the overall translation and the first downgrade translation (multiple translation results + the translation result of the first subtitle) matches the number of subtitles in the sentence; if they match, the sentence translation is considered complete, and the translation of the sentence is generated based on the overall translation result and all translation results obtained from the first downgrade translation; if they do not match, second subtitles that have not been translated are selected from each first subtitle, and the second subtitles are translated using a sentence-by-sentence translation mode. At this point, the sum of the overall translation result, the result of the first downgrade translation (the translation result of the first subtitle), and the result of the second downgrade translation (the translation result of the second subtitle) must equal the number of subtitles in the sentence, and thus the translation of the sentence can be generated based on the overall translation result, the result of the first downgrade translation, and the result of the second downgrade translation.

[0071] S103: Based on the sentence and the duration of its original subtitle, perform minimum deviation segmentation on the translation of the sentence to obtain the sentence segmentation scheme; wherein, the segmentation scheme includes the translation fragments of each subtitle corresponding to the sentence.

[0072] In the specific execution step S103, the translation of the sentence can be dubbed to obtain the dubbed translation of the sentence; the translated word sequence and its timestamp sequence of the dubbed translation of the sentence can be generated, and the translation of the sentence can be segmented with minimum deviation according to the original subtitle sentence length, the translated word sequence and the timestamp sequence to obtain the sentence segmentation scheme.

[0073] In some embodiments, TTS technology can be used to first dub the translated sentence, obtaining the initial translated dubbing and its duration. Then, based on the original subtitle duration and the initial translated dubbing duration, it is determined whether the translated sentence needs optimization (compression or expansion). If not, the initial translated dubbing and its duration can be used as the translated dubbing and its duration. If so, the translated sentence is optimized first, so that the optimized translation can be used to generate the translated dubbing and its duration, ensuring that the overall dubbing duration of the translated sentence falls within a reasonable range.

[0074] Optionally, the process of dubbing the translated sentence to obtain the dubbed translation can be as follows: obtaining a terminology set for the sentence; wherein, the terminology set includes multiple terms contained in the sentence; performing speech synthesis on the translated sentence to obtain the initial dubbed translation and its initial dubbed translation duration; calculating the duration ratio of the sentence based on the initial dubbed translation duration and the duration of the original subtitle sentence; if the duration ratio is within a preset range, using the initial dubbed translation and its initial dubbed translation duration as the dubbed translation and its duration; if the duration ratio is not within a preset range, optimizing the translated sentence using the terminology set; updating the translated sentence using the optimized translation, and returning to the step of performing speech synthesis on the translated sentence to obtain the initial dubbed translation and its initial dubbed translation duration.

[0075] In the embodiments of this application, research has revealed that due to differences in expression habits and phonetic features among different languages, the actual dubbing duration of the translated sentence often deviates from the duration of the original subtitle. If the sentence is directly segmented, the duration of each subtitle may exceed a reasonable range, requiring significant speed adjustments or repeated text adjustments to match the duration, thus compromising the integrity and naturalness of the translation. The existing technology adopts a single subtitle optimization strategy, which performs text regularization on each segmented subtitle entry. However, this method has the following limitations: (1) The sentence of a single subtitle is incomplete, and the sentence integrity is easily destroyed after optimization; (2) Names, place names, and professional terms may be mistakenly changed during the optimization process; (3) Adjustment is made once based on the target word count, but the syllables of different words are very different, and the actual dubbing time cannot be accurately reflected based solely on the word count estimation. In response to the above problems, this application proposes a sentence-level duration iterative optimization mechanism, which includes three core elements: (1) Sentence-level optimization strategy: optimize first and then segment to avoid destroying the sentence integrity; (2) Terminology locking constraint: retain all terms in the sentence during the optimization process; (3) Closed-loop iteration based on TTS feedback: ensure that the dubbing time of the translated sentence falls precisely within the target range through multiple iterations of "text optimization → TTS verification → duration evaluation → optimization again".

[0076] Specifically, based on the length of the sentence and its original subtitle, the translated text of the sentence is dubbed, and the specific process of obtaining the dubbed translation of the sentence is as follows:

[0077] (1) Initialize parameters; specifically, obtain the subtitle index set I based on the mapping relationship between sentences and the original subtitle index set. S ={i1,i2,......,i n} and the duration of the original subtitles It can also obtain the acceptable range of a pre-set duration ratio. ([Default α=0.9, β=1.1], meaning that precise matching can be achieved through subsequent fine-tuning from 0.9 to 1.1 times speed). Where n is the total number of subtitles in sentence S;

[0078] (2) Construction of term locking constraints; specifically, extracting the term set T containing all terms in sentence S from the context and term dictionary. S ={term1,term2,...,term m}; where the terms in the term set must be completely retained during the translation optimization process, and the retention conditions are shown in formula (2);

[0079] (2)

[0080] In formula (2), S is the original sentence, S' is the translation of the optimized sentence, and T... s It is a set of terms extracted from sentence S.

[0081] It should be noted that, as can be seen from formula (2), if a term is both a term in the term set and a term in the translation of the optimized sentence, then TermRetention is 1, otherwise it is 0; if the TermRetention of a term is 1, then it can be determined that the term will be retained during the optimization of sentence S; if the TermRetention of a term is 0, then it can be determined that the term will not be retained during the optimization of sentence S.

[0082] (3) Iterative optimization loop based on TTS feedback; specifically, the upper limit of the number of iterations is set to H.

[0083] (4) TTS dubbing and duration assessment; specifically, using TTS technology (using the standard speaking speed of the target language) to translate the sentence S K Perform TTS dubbing to obtain the initial translated dubbing of the sentence and its initial dubbing duration (actual dubbing duration). ), and calculate the sentence duration ratio based on the initial translation dubbing duration and the original subtitle duration, as shown in formula (3); where the target language is the language used when dubbing the translation of the sentence, such as Mandarin, English, Thai, etc.

[0084] (3)

[0085] In formula (3), This is the ratio of sentence duration. The actual dubbing duration of the sentence. is the original subtitle duration of the sentence, and K is the current iteration number.

[0086] (5) Convergence judgment; determine whether the duration ratio of the currently calculated sentence is within an acceptable range. If it is located within, it is determined that the translation of the sentence does not need to be optimized, and the initial translation dubbing and its initial translation dubbing duration can be directly used as the translation dubbing and its translation dubbing duration of the sentence; if it is not located within, it can be determined whether the current iteration number has reached the upper limit of the iteration number. If it has, the initial translation dubbing and its initial translation dubbing duration of the sentence can be directly used as the translation dubbing and its translation dubbing duration of the sentence; if it has not reached, the iteration number is controlled +1, and the translation of the sentence is optimized, that is, (6) is executed.

[0087] (6) Translation optimization: Calculate the target word count of the sentence, where the formula for calculating the target word count is shown in formula (4);

[0088] (4)

[0089] in, For the target word count, L current ρ is the current word count (the number of words in the translated sentence), γ is the safety factor (default 0.95), sign(ρ k -1) is 1 during compression and -1 during expansion; since the number of syllables varies greatly among different words, estimation based solely on the number of words cannot accurately reflect the actual dubbing duration. Introducing a safety factor here can reduce the number of iterations and improve optimization efficiency.

[0090] (7) Optimize the translation of the sentence using a large language model based on the target word count and term set, and update the current translation of the sentence using the optimized translation, so as to return to execution (4) and re-evaluate the dubbing and duration of the sentence translation until the duration ratio of the sentence is within the acceptable range (preset range), and output S. opt =S K Or, when the number of optimization iterations reaches the upper limit of iteration count H, output S. opt =SH , of which S opt For the final translation of the sentence, S H The translation of the sentence corresponding to the upper limit of the iteration count H; finally, S opt The corresponding initial translation and dubbing duration are used as the translation and dubbing duration of the sentence.

[0091] The process of optimizing the translation of a sentence using a large language model based on the target word count and term set is as follows: Optimize (compress or expand) the translation of a sentence with controlled length using a large language model. The optimization objective and the constraint are consistent, as shown in formula (5):

[0092] (5)

[0093] Where L(S') is the text length function, θ sim is the semantic similarity threshold (default 0.85), S' is the translated sentence after optimization, S0 is the translated sentence, and SemanticSimilarity is the semantic similarity value between S' and S0.

[0094] As can be seen, this application locks terms under the constraint of the original subtitle sentence duration and performs sentence-level duration iteration optimization in combination with TTS feedback, so that the optimized translation can be directly used on the timeline, avoiding the sentence being fragmented and resulting in semantic incoherence, and also ensuring that

[0095] In this embodiment, after optimizing the duration of the sentence translation to obtain the final translation and its dubbing, since the final output needs to maintain the consistency between the number of pre-translation subtitles and the number of subtitle translation results in the final translation, the final translation of the sentence needs to be divided into n translation segments (n is the number of original subtitles, i.e., the number of pre-translation subtitles), so that each translation segment corresponds to one original subtitle, and the dubbing duration of the translation segment is as close as possible to the duration of the corresponding original subtitle, i.e., the duration ratio of the dubbing duration to the duration of the corresponding original subtitle sentence is within a preset range, and the segmentation point of each translation segment is as natural as possible to ensure semantic integrity.

[0096] Optionally, after obtaining the dubbing duration of the sentence, a timestamp prediction model can be used to predict the translated word sequence and timestamp sequence of the dubbing. A dynamic programming algorithm is then used to perform minimum deviation segmentation of the sentence's translation based on the original subtitle duration, translated word sequence, and timestamp sequence to obtain a segmentation scheme for the sentence. The segmentation scheme includes the translated segments of each subtitle.

[0097] It should be noted that the timestamp prediction model can be a Timestamp Prediction Model, but this embodiment of the application does not limit it.

[0098] It should also be noted that using the timestamp prediction model to predict the translated word sequence and timestamp sequence of the dubbing of a sentence not only eliminates the need for forced alignment between the source language audio and the target language audio (dubbing), but also has low computational cost and low latency.

[0099] In some embodiments, the translated dubbing of a sentence is input into a timestamp prediction model, causing the timestamp prediction model to output a sequence of translated words {w1, w2, ..., w}. m} and the corresponding timestamp {[ ],[ ],...,[ ]}.

[0100] In this embodiment, based on the original subtitle sentence length, translated word sequence, and timestamp sequence, a dynamic programming algorithm is used to segment the translated sentence back into the original subtitle according to the principle of minimum total deviation, resulting in a segmentation scheme containing multiple translated segments of the sentence; the specific segmentation process is as follows:

[0101] Input to the dynamic programming algorithm: original subtitle duration (n is the number of original subtitles in the sentence), translated word sequence (m is the total number of words), timestamp list ;

[0102] Output of the dynamic programming algorithm: partitioning scheme ,in Indicates the range of translated words corresponding to the j-th subtitle ( ), p is the cumulative word position index;

[0103] The optimization objective of the dynamic programming algorithm in the segmentation process is to minimize the deviation between the translation and the original subtitle duration, while ensuring that the ratio of the translation dubbing duration to the original subtitle duration of each subtitle segment is within a reasonable speed range ([ɑ,β], default ɑ=0.9, β=1.1); the specific optimization method is shown in formula (6):

[0104] (6)

[0105] in, The actual duration of the translated segment of the j-th subtitle. Add 1 to the end position of the (j-1)th subtitle. This is the end position of the j-th subtitle. is the original subtitle duration of the j-th subtitle. During the optimization process, a penalty function can also be added, and this penalty function is shown in Formula (7):

[0106] (7)

[0107] where is the original subtitle duration of the j-th subtitle, and M is the penalty coefficient for exceeding the range, which can ensure that all the subtitles selected preferentially are within the reasonable range of segmentation schemes. The design of this penalty function enables the dynamic programming algorithm to distinguish between "deviations within the speed-up / slow-down interval" and "deviations beyond the speed-up / slow-down interval", so as to preferentially ensure that the dubbed durations of all subtitle translation segments fall within the acceptable range in the global optimum.

[0108] Further, the process of the dynamic programming algorithm is explained as follows:

[0109] First, define the corresponding states and boundary conditions: Define dp[i][j] as the minimum total penalty value for allocating the first i words in the translation of the sentence to the first j subtitles; Boundary conditions: dp[0][0]=0 (0 words match 0 subtitles), dp[i][0]= (There are words but no subtitles to match), dp[0][j]= (There are no words but there are subtitles to match).

[0110] Second, fill in the table from bottom to top: Calculate dp[i][j] in the order of i from 1 to m and j from 1 to n. For dp[i][j] of each state, enumerate the previous segmentation point k (j - 1 ≤ k < i), and allocate the j-th subtitle to the word w k+1 to w i , and take the minimum value among all possible k. Among them, the calculation method of dp[i][j] is shown in Formula (8):

[0111] (8)

[0112] where is the duration of the word w k+1 to w i , that is, the dubbed duration corresponding to the j-th subtitle; During the calculation process of Formula (8), record the segmentation point k (the best segmentation point) when reaching the minimum value, and store the best segmentation point k in split[i][j]. The calculation method of this split[i][j] is shown in Formula (9):

[0113] (9)

[0114] Here, split[i][j] indicates that when dp[i][j] is optimal, the translated segment of the first j-1 subtitles uses the first split[i][j] words, that is, the translated segment of the j-th subtitle starts from the split[i][j]+1 word. When calculating dp[i][j], the value of dp[k][j-1] has been calculated and stored beforehand, and can be directly retrieved from the table to avoid redundant calculations. For example, when dp[i][j] is dp[5][3], the calculated split[5][3]=3, which means that to minimize the total penalty of "distributing the first 5 words to the first 3 subtitles", the optimal splitting method is: the first 2 subtitles distribute the first 3 words, and the third subtitle distributes the remaining 4th and 5th words.

[0115] Finally, the split point is obtained by backtracking from top to bottom: after filling the table, the optimal total penalty value is dp[m][n]. Backtracking is based on the optimal substructure property of dynamic programming: the optimal solution of dp[m][n] must be composed of the optimal solutions to subproblems, so the globally optimal splitting scheme can be obtained by backtracking along the split array. The backtracking process is as follows:

[0116] First, initialize: p n =m (The last subtitle uses the last word);

[0117] Secondly, backtrack sequentially from j=n to j=2: p j-1 =split[i][j], indicating that in the optimal solution, the translated segments of the first j-1 subtitles use the first p... j-1 One word;

[0118] Finally, the sequence of split points {p1, p2, ..., p} is obtained. n}, where p j This indicates that the range of translated words corresponding to the j-th subtitle is [p j-1 +1,p j ].

[0119] S104: Dub the translated text segment of each subtitle and adjust the dubbing rate of the translated text segment of each subtitle based on the original subtitle sentence length.

[0120] In the specific execution of step S104, after obtaining the sentence segmentation scheme, for each subtitle segment in the segmentation scheme, TTS synthesis is performed on the subtitle segment to obtain the dubbing of the subtitle segment; the dubbing rate of the subtitle segment dubbing is adjusted based on the dubbing duration of the subtitle segment dubbing and the original subtitle duration.

[0121] In practical applications, for each subtitle segment in the segmentation scheme, TTS synthesis can be performed on the subtitle segment to obtain the dubbing of the subtitle segment; the ratio of the dubbing duration of the subtitle segment to the duration of the original subtitle is calculated, and the ratio is finely adjusted within the speed range of [α,β] (default α=0.9,β=1.1]) to ensure that the dubbing of the subtitle segment matches the subtitle timing precisely.

[0122] This application provides a subtitle processing method, which generates multiple sentences and their original subtitle sentence durations based on a subtitle file; wherein each sentence consists of at least one subtitle, and the original subtitle sentence duration is the sum of the original subtitle durations of all subtitles constituting the sentence; the sentences are translated to obtain the translated text; based on the sentences and their original subtitle sentence durations, the translated text of the sentences is segmented with minimum deviation to obtain a segmentation scheme; wherein the segmentation scheme includes the translated text segments of each subtitle corresponding to the sentence; the translated text segments of each subtitle are dubbed, and the dubbing rate of the dubbing of the translated text segments of each subtitle is adjusted based on the original subtitle duration. Therefore, this application first generates multiple sentences and their original subtitle durations based on the subtitle file; secondly, it translates the subtitles in the sentences to obtain complete translations of the sentences, so that the translations of the sentences can be segmented with minimal deviation. This allows the resulting sentence segmentation scheme to be directly applied on the timeline, which not only avoids sentence fragmentation and semantic incoherence, but also achieves high-precision time alignment while ensuring semantic integrity and viewing flow. This solves the problems of inconsistent terminology translation, semantic breaks, low translation efficiency, unnatural dubbing, and high rework rate in existing technologies.

[0123] Based on the subtitle processing method provided in the embodiments of this application above, correspondingly, this application provides a subtitle processing system, such as... Figure 2 As shown, the subtitle processing system includes:

[0124] The subtitle merging unit 21 is used to generate multiple sentences and their original subtitle sentence durations based on the subtitle file; wherein, a sentence consists of at least one subtitle, and the original subtitle sentence duration is the sum of the original subtitle durations of all subtitles that make up the sentence;

[0125] Translation unit 22 is used to translate sentences and obtain the translated version of the sentences;

[0126] The translation optimization and segmentation unit 23 is used to perform minimum deviation segmentation of the translation of a sentence based on the sentence and the duration of its original subtitle, and obtain a segmentation scheme for the sentence; wherein, the segmentation scheme includes the translation fragments of each subtitle corresponding to the sentence;

[0127] The translation and dubbing synthesis unit 24 is used to dub the translated text segment of each subtitle and adjust the dubbing rate of the translated text segment of each subtitle based on the original subtitle duration.

[0128] This application provides a subtitle processing system that generates multiple sentences and their original subtitle durations based on a subtitle file; wherein each sentence consists of at least one subtitle, and the original subtitle duration is the sum of the original subtitle durations of all subtitles constituting the sentence; the sentences are translated to obtain a translated text; based on the sentences and their original subtitle durations, the translated text of the sentences is segmented with minimum deviation to obtain a segmentation scheme; wherein the segmentation scheme includes a translated text segment for each subtitle corresponding to the sentence; each translated text segment of a subtitle is dubbed, and the dubbing rate of the dubbing of the translated text segment of each subtitle is adjusted based on the original subtitle duration. Therefore, this application first generates multiple sentences and their original subtitle durations based on the subtitle file; secondly, it translates the subtitles in the sentences to obtain complete translations of the sentences, so that the translations of the sentences can be segmented with minimal deviation. This allows the resulting sentence segmentation scheme to be directly applied on the timeline, which not only avoids sentence fragmentation and semantic incoherence, but also achieves high-precision time alignment while ensuring semantic integrity and viewing flow. This solves the problems of inconsistent terminology translation, semantic breaks, low translation efficiency, unnatural dubbing, and high rework rate in existing technologies.

[0129] Optionally, a subtitle merging unit is generated based on the subtitle file, containing multiple sentences and their original subtitle durations. This is specifically used for:

[0130] Obtain the subtitle file, which contains multiple subtitles;

[0131] The subtitle file is traversed using a preset sliding window, and the sentence end positions in the subtitle text within the preset sliding window are identified by a large language model until the entire subtitle file is traversed, resulting in multiple sentence end positions.

[0132] The subtitles are merged according to the end position of each sentence to obtain multiple sentences;

[0133] Generate a set of original subtitle indices for the sentence, and generate the original subtitle duration based on the set of original subtitle indices.

[0134] Optionally, a translation unit is used to translate the sentence and obtain its translated version, specifically for:

[0135] The translation model is used to translate each subtitle of the sentence, resulting in multiple translation results;

[0136] If the number of multiple translation results is not equal to the number of subtitles in the sentence, the first subtitle that has not been translated is split, and the split result of the first subtitle is translated using the translation model to obtain the translation result of the first subtitle;

[0137] If the sum of the number of multiple translation results and the number of translation results of the first subtitle is not equal to the number of subtitles, the sentence-by-sentence translation mode is used to translate the second subtitles that have not been translated in the sentence, and the translation result of the second subtitle is obtained;

[0138] The translation of the sentence is generated based on multiple translation results, the translation result of the first subtitle, and the translation result of the second subtitle.

[0139] Optionally, based on the sentence and the duration of its original subtitle, the translation of the sentence is segmented with minimum deviation to obtain a translation optimization and segmentation unit for the sentence segmentation scheme, specifically used for:

[0140] Dubbing the translated sentence produces a dubbed version of the sentence.

[0141] The system generates a sequence of translated words and a sequence of timestamps for the dubbing of the sentence. Based on the original subtitle duration, the sequence of translated words, and the sequence of timestamps, the system performs minimum deviation segmentation of the sentence to obtain a segmentation scheme.

[0142] Optionally, the translated sentence is dubbed to obtain a translation optimization and segmentation unit for dubbing the translated sentence, specifically used for:

[0143] Get the term set of the sentence; where the term set includes multiple terms contained in the sentence;

[0144] The translation of the sentence is processed by speech synthesis to obtain the initial translation dubbing and its duration.

[0145] Calculate the sentence length ratio based on the initial dubbing duration and the original subtitle duration;

[0146] If the duration ratio is within the preset range, the initial translation of the sentence and its initial translation duration will be used as the translation of the sentence and its translation duration.

[0147] If the duration ratio is not within the preset range, the translation of the sentence is optimized using a terminology set;

[0148] The optimized translation is used to update the sentence's translation, and then the process of performing speech synthesis on the sentence's translation is returned to obtain the initial translation dubbing and its initial dubbing duration.

[0149] Optionally, the translated word sequence and its timestamp sequence of the dubbed version of the sentence are generated, and the translation of the sentence is segmented with minimum deviation based on the original subtitle sentence length, the translated word sequence, and the timestamp sequence, resulting in a translation optimization and segmentation unit for the sentence segmentation scheme, specifically used for:

[0150] The timestamp prediction model is used to predict the translated word sequence and timestamp sequence of the dubbing of a sentence.

[0151] A dynamic programming algorithm is used to perform minimum deviation segmentation of the sentence translation based on the original subtitle length, the translated word sequence, and the timestamp sequence, resulting in a sentence segmentation scheme.

[0152] This application also provides a storage medium storing program instructions, which, when loaded and executed by a processor, implement any of the above-described subtitle processing method embodiments.

[0153] This application also provides an electronic device, such as Figure 3 As shown, the device includes a processor 301 and a memory 302, which are connected via a bus; the memory stores program instructions; the processor calls the program instructions in the memory to execute any of the above-described subtitle processing method embodiments.

[0154] The processor mentioned in this article can be the terminal's CPU, an integrated MCU within the terminal, or a combination of a CPU and an MCU. Furthermore, the processor contains a kernel that retrieves the corresponding program from memory; one or more kernels can be configured.

[0155] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0158] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0159] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A subtitle processing method, characterized in that, The method includes: Multiple sentences and their original subtitle durations are generated based on the subtitle file; wherein, each sentence consists of at least one subtitle, and the original subtitle duration is the sum of the original subtitle durations of all subtitles that constitute the sentence; The sentence is translated to obtain its translated form; Based on the sentence and the duration of its original subtitle, the translation of the sentence is segmented with minimum deviation to obtain a segmentation scheme for the sentence; wherein, the segmentation scheme includes the translated segments of each subtitle corresponding to the sentence; The translated text segment of each subtitle is dubbed, and the dubbing rate of the dubbing of the translated text segment of each subtitle is adjusted based on the original subtitle duration; The sentence is translated to obtain the translated text, including: The translation model is used to translate each subtitle of the sentence, resulting in multiple translation results; If the number of multiple translation results is not equal to the number of subtitles in the sentence, the first subtitle that has not been translated is split, and the split result of the first subtitle is translated using the translation model to obtain the translation result of the first subtitle; If the sum of the number of multiple translation results and the number of translation results of the first subtitle is not equal to the number of subtitles, the sentence-by-sentence translation mode is used to translate the second subtitle that has not been translated in the sentence, and the translation result of the second subtitle is obtained; The translation of the sentence is generated based on the multiple translation results, the translation result of the first subtitle, and the translation result of the second subtitle.

2. The method according to claim 1, characterized in that, Multiple sentences are generated based on the subtitle file, along with the duration of their original subtitle sentences, including: Obtain a subtitle file, wherein the subtitle file includes multiple subtitles; The subtitle file is traversed using a preset sliding window, and the sentence end positions in the subtitle text within the preset sliding window are identified by a large language model until the entire subtitle file is traversed, resulting in multiple sentence end positions. The subtitles are merged according to the end position of each sentence to obtain multiple sentences; Generate a set of original subtitle indexes for the sentence, and generate the original subtitle duration for the sentence based on the set of original subtitle indexes.

3. The method according to claim 1, characterized in that, Based on the length of the sentence and its original subtitle, the translation of the sentence is segmented using minimum deviation to obtain a segmentation scheme for the sentence, including: The translated version of the sentence is dubbed to obtain the dubbed translation of the sentence; Generate the translated word sequence and its timestamp sequence for the dubbing of the sentence, and perform minimum deviation segmentation of the translation of the sentence based on the original subtitle sentence duration, translated word sequence and timestamp sequence to obtain the segmentation scheme of the sentence.

4. The method according to claim 3, characterized in that, The translation of the sentence is dubbed to obtain the dubbed translation of the sentence, including: Obtain the term set of the sentence; wherein the term set includes multiple terms contained within the sentence; The translation of the sentence is processed by speech synthesis to obtain the initial translation dubbing of the sentence and the duration of the initial translation dubbing; Calculate the length ratio of the sentence based on the initial dubbing duration and the original subtitle duration; If the duration ratio is within a preset range, the initial translated dubbing of the sentence and its initial translated dubbing duration will be used as the translated dubbing of the sentence and its translated dubbing duration. If the duration ratio is not within a preset range, the translation of the sentence is optimized using the terminology set; The optimized translation is used to update the translation of the sentence, and then the process of performing speech synthesis on the translation of the sentence is returned to obtain the initial translation dubbing of the sentence and its initial dubbing duration.

5. The method according to claim 4, characterized in that, Generate the translated word sequence and its timestamp sequence for the dubbing of the sentence, and perform minimum deviation segmentation of the translated sentence based on the original subtitle sentence length, the translated word sequence, and the timestamp sequence to obtain the sentence segmentation scheme, including: The timestamp prediction model is used to predict the translated word sequence and timestamp sequence of the dubbing of the sentence. A dynamic programming algorithm is used to perform minimum deviation segmentation of the translation of the sentence based on the original subtitle length, the translated word sequence, and the timestamp sequence, resulting in a segmentation scheme for the sentence.

6. A subtitle processing system, characterized in that, The system includes: A subtitle merging unit is used to generate multiple sentences and their original subtitle sentence durations based on a subtitle file; wherein, each sentence consists of at least one subtitle, and the original subtitle sentence duration is the sum of the original subtitle durations of all subtitles constituting the sentence; A translation unit is used to translate the sentence to obtain a translated version of the sentence; The translation optimization and segmentation unit is used to perform minimum deviation segmentation of the translation of the sentence based on the sentence and the duration of its original subtitle, to obtain a segmentation scheme for the sentence; wherein, the segmentation scheme includes the translated segments of each subtitle corresponding to the sentence; The translation and dubbing synthesis unit is used to dub the translated text segment of each subtitle and adjust the dubbing rate of the translated text segment of each subtitle based on the original subtitle sentence length; The translation unit is specifically used for: The translation model is used to translate each subtitle of the sentence, resulting in multiple translation results; If the number of multiple translation results is not equal to the number of subtitles in the sentence, the first subtitle that has not been translated is split, and the split result of the first subtitle is translated using the translation model to obtain the translation result of the first subtitle; If the sum of the number of multiple translation results and the number of translation results of the first subtitle is not equal to the number of subtitles, the sentence-by-sentence translation mode is used to translate the second subtitle that has not been translated in the sentence, and the translation result of the second subtitle is obtained; The translation of the sentence is generated based on the multiple translation results, the translation result of the first subtitle, and the translation result of the second subtitle.

7. The system according to claim 6, characterized in that, A subtitle merging unit is generated based on the subtitle file, containing multiple sentences and their original subtitle durations. Specifically, it is used for: Obtain a subtitle file, wherein the subtitle file includes multiple subtitles; The subtitle file is traversed using a preset sliding window, and the sentence end positions in the subtitle text of the preset sliding window are identified by a large language model until the subtitle file is completely traversed, resulting in multiple sentence end positions. The subtitles are merged according to the end position of each sentence to obtain multiple sentences; Generate a set of original subtitle indexes for the sentence, and generate the original subtitle duration for the sentence based on the set of original subtitle indexes.

8. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the subtitle processing method as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for performing the subtitle processing method as described in any one of claims 1-5.

Images