Oral subtitle alignment method and device, equipment, storage medium and program product

Abstract

The invention discloses an oral subtitle alignment method and device, equipment, a storage medium and a program product, and relates to the technical field of subtitle alignment, and the method comprises the steps: carrying out the segmentation processing of oral characters, sequentially inputting the segmented oral characters into a voice generation model, and carrying out the voice generation processing, and obtaining the audio data corresponding to each segment of oral characters; splicing each segment of audio data to obtain an oral broadcast audio, and determining a start-end time point of each segment of oral broadcast text; dividing the oral broadcast characters according to a subtitle division rule to obtain a subtitle set; determining an initial access point of each subtitle in the subtitle set based on the total duration of the oral broadcast audio and the subtitle length of each subtitle in the subtitle set; correcting the initial access point of each subtitle based on the start-end time point of each segment of oral broadcast text, determining the subtitle access point of each subtitle, and obtaining standard subtitle data; and generating subtitles in the corresponding oral broadcast file based on the standard subtitle data. The method and the device have the technical effect of improving the matching precision of the mouth broadcast subtitles.

Description

Technical Field

[0001] This application relates to the field of subtitle alignment technology, and in particular to methods, apparatus, equipment, storage media, and program products for aligning spoken subtitles. Background Technology

[0002] With the rapid development of content industries such as short videos, online education, and live-streaming e-commerce, accurate synchronization of audio and subtitles in spoken-word videos has become a crucial aspect of content production. In spoken-word scenarios, subtitles not only need to be semantically coherent but also require their display to be as synchronized as possible with the audio to achieve a natural and smooth viewing experience.

[0003] Currently, with the development of artificial intelligence technology, the generation of spoken audio using speech generation models is becoming increasingly common. Common subtitle generation techniques mainly rely on ASR (Automatic Speech Recognition) for whole-segment audio recognition, then generate subtitle timelines through an average distribution method. However, when using speech generation models to generate spoken audio, issues arise such as inconsistencies between the speech generation model's segmentation rules for spoken text and subtitle division, and uncontrollable pause durations in the generated speech. This leads to shortcomings in the average distribution method for determining subtitle timelines, including low matching accuracy, misaligned subtitle in / out points, and inaccurate time allocation within sentences. Therefore, a subtitle alignment technology specifically designed for spoken audio files generated by speech generation models is urgently needed. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, device, storage medium, and program product for aligning spoken subtitles, aiming to solve the technical problem of low matching accuracy in the alignment of spoken subtitles in related technologies.

[0005] To achieve the above objectives, this application proposes a method for aligning spoken subtitles, the method comprising: The spoken text is segmented and processed, and then sequentially input into the speech generation model for speech generation processing to obtain the audio data corresponding to each segment of spoken text. The audio data segments are spliced ​​together to obtain the spoken audio, and the start-end time points of each spoken text segment are determined. The spoken text is divided according to the subtitle segmentation rules to obtain a subtitle set; Based on the total duration of the spoken audio and the length of each subtitle in the subtitle set, determine the initial entry and exit points of each subtitle in the subtitle set; Based on the start-end time points of each spoken text segment, the initial entry and exit points of each subtitle are corrected to determine the entry and exit points of each subtitle and obtain standard subtitle data. Subtitles are generated in the corresponding audio file based on standard subtitle data.

[0006] In one embodiment, the step of segmenting the spoken text and then sequentially inputting it into a speech generation model for speech generation processing to obtain the audio data corresponding to each segment of spoken text includes: Based on the type of the voice generation model invoked, the segmentation processing method of the spoken text is determined, and multiple segments of text to be processed are obtained. Each segment of text to be processed is input into the speech generation model in sequence to obtain the corresponding audio data, and the start time of each audio data is recorded. The audio data is concatenated into a spoken audio file according to the input order of the text to be processed.

[0007] In one embodiment, the step of dividing the spoken text according to a preset subtitle division rule to obtain a subtitle set includes: Based on the line breaks in the spoken text, the spoken text is first split to obtain the first split result; Based on the designed length of the subtitles, the first splitting result is further split to obtain the second splitting result; Remove the punctuation marks at the end of each subtitle in the second split result to obtain the subtitle set.

[0008] In one embodiment, the step of determining the initial entry and exit points of each subtitle in the subtitle set based on the total duration of the spoken audio and the subtitle length of each subtitle in the subtitle set includes: Determine the length of each subtitle in the subtitle set and the total length of all subtitles; Iterate through each subtitle in the set and determine the duration ratio of the subtitle based on the ratio of the subtitle length to the total subtitle length; Based on the duration ratio and the total duration of the spoken audio, the initial entry and exit points of each subtitle in the subtitle set are determined.

[0009] In one embodiment, the steps of correcting the initial entry and exit points of each subtitle based on the start-end time points of each spoken text segment, determining the entry and exit points of each subtitle, and obtaining standard subtitle data include: Based on the start-end time points of the spoken text and the text length of each segment of spoken text, the start-end time of each character in each segment of spoken text is determined, and a single-character duration array is obtained. For each character in the single-character duration array, take one character forward and one character backward to form a three-character group, determine the start-end time point of each three-character group, and generate a three-character duration array; For each subtitle in the subtitle set, a sampling window size of three characters and a sampling sliding step of one character are used to sequentially extract multiple groups of characters to be matched from the subtitle; wherein the last character in the last group of characters to be matched is the same as the last character in the corresponding subtitle. For each character group to be matched in any subtitle, find a three-character group that matches completely from the three-character duration array, and count the number of character groups to be matched that match the three-character group in each subtitle as the number of collisions; For each subtitle in the subtitle set, the hit rate is determined based on the ratio of the number of collisions to the subtitle length; Subtitles that meet the accuracy requirements are corrected to obtain standard subtitle data.

[0010] In one embodiment, the step of correcting subtitles that meet the hit rate requirements to obtain standard subtitle data includes: For each subtitle in the subtitle set, if the subtitle is a perfect match, the in-time of the subtitle is modified to the in-time of the first matched three-character group in the subtitle, which is used as the initial in-time; the out-time of the subtitle is modified to the out-time of the last matched three-character group in the subtitle, which is used as the corrected out-time. If the subtitle is not a complete match, the in-time of the first three-character group matched in the subtitle is modified based on the character distance between the first character of the first matched character group and the first character of the subtitle, and is used as the initial in-time. Similarly, the out-time of the last matched character group is modified based on the character distance between the last character of the last matched character group and the last character of the subtitle, and is used as the correction out-time. If the initial in-time is less than the out-time of the previous subtitle in the subtitle set, the initial in-time is modified to the out-time of the previous subtitle to obtain the corrected in-time. Standard subtitle data is determined based on the correction in-point time and correction out-point time.

[0011] Secondly, to achieve the above objectives, this application further provides a spoken subtitle alignment device, the device comprising: The audio generation module is used to process the spoken text into segments and input them sequentially into the speech generation model for speech generation processing, so as to obtain the audio data corresponding to each segment of spoken text. The audio splicing module is used to splice together audio data from different segments to obtain spoken audio and determine the start and end time points of each spoken text segment. The subtitle segmentation module is used to segment the spoken text according to subtitle segmentation rules to obtain a subtitle set; The subtitle alignment module is used to determine the initial in / out point of each subtitle in the subtitle set based on the total duration of the spoken audio and the length of each subtitle in the subtitle set; The subtitle correction module is used to correct the initial entry and exit points of each subtitle based on the start-end time points of each spoken text segment, determine the entry and exit points of each subtitle, and obtain standard subtitle data. The subtitle generation module is used to generate subtitles from the corresponding audio file based on standard subtitle data.

[0012] Thirdly, to achieve the above objectives, this application further provides a voice-over subtitle alignment device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the above-described voice-over subtitle alignment method.

[0013] Fourthly, to achieve the above objectives, this application further provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above-described audio subtitle alignment method.

[0014] Fifthly, to achieve the above objectives, this application further provides a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the steps of the above-described method for aligning spoken subtitles.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application generates segmented audio from spoken text using a speech generation model and constructs a fine-grained time alignment mechanism based on the actual start and end times of the generated segments. This achieves automatic correction of the initial timeline of the subtitles, effectively overcoming the problem of low matching accuracy of the average-distribution subtitle timeline caused by inconsistent segment divisions and uncontrollable pause durations in the speech generation model. By utilizing a three-character group matching statistical hit rate method, this application can accurately locate the actual landing point of the subtitles in the audio at the character level, ensuring that the subtitle entry and exit points are highly consistent with the speech rhythm, improving the accuracy and naturalness of subtitle synchronization, thereby guaranteeing the viewing experience of spoken videos. It is particularly suitable for subtitle alignment scenarios using spoken audio automatically generated by a speech generation model. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for aligning spoken subtitles in one embodiment of this application.

[0019] Figure 2 This is a schematic diagram illustrating the effect of the voice-over subtitle alignment method in a specific implementation example of this application.

[0020] Figure 3 This is a schematic diagram of the structure of the audio-visual text alignment device of this application.

[0021] Figure 4 This is a schematic diagram of the audio-visual text alignment device of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application is as follows: This application segments the spoken text according to the rules of the speech generation model and generates audio segment by segment, records the actual start and end times of each segment, and on this basis, performs an initial timeline allocation of the global subtitles according to their length proportions. Then, it uses character-level and three-character group-level time alignment methods to accurately correct the initial entry and exit points of the subtitles, and finally obtains a standard subtitle timeline that highly matches the actual spoken audio, thereby achieving high-precision and automated subtitle alignment in generative spoken audio scenarios.

[0026] Based on this, this application provides a method for aligning spoken subtitles, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the audio-visual subtitle alignment method of this application.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or a voice-over subtitle alignment device capable of performing the above functions. The following description uses a voice-over subtitle alignment device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] In this embodiment, the method for aligning spoken subtitles includes steps S10 to S60: Step S10: The spoken text is segmented and processed, and then sequentially input into the speech generation model for speech generation processing to obtain the audio data corresponding to each segment of spoken text.

[0029] Step S20: Concatenate the audio data segments to obtain the spoken audio and determine the start-end time points of each spoken text segment.

[0030] It's important to clarify that "speech generation" refers to the input text used to generate the audio for broadcast, i.e., the text content that is intended to be read aloud. A speech generation model is a model used to convert text into speech (such as TTS, voiceprint synthesis models, etc.). Different models handle text segmentation, pauses, and speech rate differently. Understandably, speech generation models usually have certain requirements regarding the upper and lower limits of the input text's character count, which means that the segmentation method for the text input to the speech generation model differs from the segmentation method for subtitles.

[0031] In one feasible implementation, step S10 includes steps A10 to A30: Step A10: Based on the type of the invoked speech generation model, determine the segmentation processing method for the spoken text and obtain multiple segments of text to be processed.

[0032] Step A20: Input each segment of text to be processed into the speech generation model in sequence to obtain the corresponding audio data and record the start time of each audio data.

[0033] Step A30: Concatenate the audio data into a spoken audio file according to the input order of the text to be processed.

[0034] For example, a spoken text is first input, and the spoken text is then split into segments depending on the model being called. Understandably, different models have different requirements; some models can split the text into finer segments, while others limit the minimum and maximum input text lengths. Here, we don't need to worry about how many segments the text is split into; we just need to minimize the number of characters in each segment to improve the speech generation effect and the accuracy of subsequent matching.

[0035] The segmented spoken text is sequentially sent to a large model for speech generation. After audio data is generated, the audio duration and the corresponding spoken text are recorded. Using either concurrent or serial processing, the audio segments are merged into a single complete spoken audio file after all segments have been generated. The start-end time points of each audio segment within the spoken audio are determined, thereby identifying the start-end time points of the corresponding spoken text, thus forming a segmented spoken text data group.

[0036] In this example, the spoken text is: "As educators, we are not only imparting knowledge, but also sowing hope and illuminating the future. We are not alone on this path. Every educator's efforts are part of changing the future. Even when feeling down and tired, recalling these beautiful moments will fill us with renewed strength to continue. We firmly believe that the power of education is boundless; it can change everything." After speech generation, the corresponding spoken text segment data groups can be found in Table 1, the spoken text start-end timetable.

[0037] Table 1: Start-End Schedule of Voice-Over Text Step S30: Divide the spoken text according to the subtitle division rules to obtain a subtitle set.

[0038] Step S40: Based on the total duration of the spoken audio and the length of each subtitle in the subtitle set, determine the initial entry and exit points of each subtitle in the subtitle set.

[0039] Step S50: Based on the start-end time points of each spoken text segment, the initial entry and exit points of each subtitle are corrected to determine the entry and exit points of each subtitle and obtain standard subtitle data.

[0040] It should be noted that the entry and exit points of subtitles refer to the specific times when the subtitles appear and disappear.

[0041] Specifically, the spoken text is first divided into multiple subtitles according to the preset subtitle segmentation rules. Then, based on the total duration of the entire spoken audio segment and the length ratio of each subtitle in the text (i.e., the average distribution method), an initial time range is assigned to each subtitle to obtain the initial entry and exit points. Combining the actual start and end times of each segment of text during the speech generation process, these initial time points are further corrected. Through character-level matching, the actual landing point of the subtitle in the audio is accurately located, thereby obtaining a standard subtitle timeline that is highly consistent with the spoken audio.

[0042] In a feasible implementation, step S30 includes steps B10 to B30: Step B10: Based on the line breaks in the spoken text, the spoken text is split for the first time to obtain the first split result.

[0043] Step B20: According to the designed length of the subtitle, the first split result is further split to obtain the second split result.

[0044] Step B30: Remove the punctuation marks at the end of each subtitle in the second split result to obtain the subtitle set.

[0045] Step S40 includes steps C10 to C30: Step C10: Determine the length of each subtitle in the subtitle set and the total length of all subtitles.

[0046] Step C20: Iterate through each subtitle in the set and determine the duration ratio of the subtitle based on the ratio of the subtitle length to the total subtitle length.

[0047] Step C30: Based on the duration ratio and the total duration of the spoken audio, determine the initial entry and exit points of each subtitle in the subtitle set.

[0048] Step S50 includes steps D10 to D60: Step D10: Based on the start-end time points of the spoken text and the text length of each segment of spoken text, determine the start-end time of each character in each segment of spoken text, and obtain a single-character duration array.

[0049] Step D20: For each character in the single-character duration array, take one character forward and one character backward to form a three-character group, determine the start-end time point of each three-character group, and generate a three-character duration array.

[0050] Step D30: For each subtitle in the subtitle set, take three characters as the sampling window size and one character as the sampling sliding step size, and extract multiple groups of characters to be matched from the subtitle in sequence; wherein, the last character in the last group of characters to be matched is the same as the last character in the corresponding subtitle.

[0051] Step D40: For each character group to be matched in any subtitle, find a three-character group that matches perfectly from the three-character duration array, and count the number of character groups to be matched that match in each subtitle as the number of collisions.

[0052] Step D50: For each subtitle in the subtitle set, determine the subtitle hit rate based on the ratio of the number of collisions to the subtitle length.

[0053] Step D60: Correct the subtitles that meet the hit rate requirements to obtain standard subtitle data.

[0054] Step D60 includes steps D61 to D64: Step D61: For each subtitle in the subtitle set, if the subtitle is a perfect match, modify the in-time of the subtitle to the in-time of the first matched three-character group in the subtitle as the initial in-time; modify the out-time of the subtitle to the out-time of the last matched three-character group in the subtitle as the corrected out-time.

[0055] Step D62: If the subtitle is not a complete match, then based on the character distance between the first character of the first matched character group and the first character in the subtitle, modify the in-point time of the first matched three-character group of the subtitle as the initial in-point time; based on the character distance between the last character of the last matched character group and the last character in the subtitle, modify the out-point time of the last matched three-character group of the subtitle as the correction out-point time.

[0056] Step D63: If the initial in-point time is less than the out-point time of the previous subtitle in the subtitle set, then modify the initial in-point time to the out-point time of the previous subtitle to obtain the corrected in-point time.

[0057] Step D64: Determine the standard subtitle data based on the correction in-point time and correction out-point time.

[0058] Specifically, first determine the total duration of the spoken audio. Then, use punctuation or semantics to finely break down the spoken text. You can start by breaking it down by line breaks, and then break each sentence down further if it exceeds the designed length (e.g., 15 characters). The final result is a set of finely broken-down subtitles. It is important to note that if there are punctuation marks at the end of the split subtitles, they will be removed to optimize display. Therefore, the subtitle length needs to be recalculated after fine-tuning.

[0059] After obtaining the split subtitle set, determine the total length of all subtitles in the subtitle set, traverse each subtitle in the subtitle set, determine the length of each subtitle, and calculate the subtitle length / total subtitle length * total audio duration, round down to obtain the subtitle duration of each subtitle in the subtitle set.

[0060] Based on the duration and order of each subtitle, the initial in-point and out-point of each subtitle are obtained, resulting in a set of subtitles containing the initial in-point and out-points.

[0061] In this example, the subtitle set can be represented as [{"inPoint": 0,"outPoint": 1336,"content": "As educators"},{"inPoint": 1336,"outPoint": 4009,"content": "We are not only imparting knowledge"},{"inPoint": 4009,"outPoint": 7216, "content": "We are also sowing hope and illuminating the future"},{"inPoint": 7216,"outPoint": 10156,"content": "On this path, we are not alone"},{"inPoint": 10156,"outPoint": 12561,"content": "The efforts of every educator"},{"inPoint": 12561,"outPoint": 15234,"content": "All are part of changing the future"},{"inPoint": 15234,"outPoint": 18174,"content": "Even when feeling down and tired"},{"inPoint": {"inPoint": 20847,"content": "Recalling these beautiful moments"},{"inPoint": 20847,"outPoint": 24321,"content": "Then we will be filled with strength again and continue to move forward"},{"inPoint": 24321,"outPoint": 28063,"content": "We firmly believe that the power of education is infinite"},{"inPoint": 28063,"outPoint": 29934,"content": "It can change everything"}).

[0062] Subsequently, based on the information in the spoken text segmentation data set, a single-word duration array is generated. Specifically, during speech generation, the speech generation model has a certain pause time for punctuation marks. Therefore, for the spoken text segmentation data set, it is necessary to distinguish between two types: ordinary pronunciation characters (text) and punctuation marks. This step requires calculating the number of punctuation marks in each sentence in advance. By removing the punctuation marks, the remaining ordinary pronunciation characters are calculated, and the duration of this sentence is evenly distributed by these ordinary pronunciation characters. Here, taking the example that a punctuation mark occupies 10 milliseconds when the speech generation model generates audio. If a 30-word sentence has 4 punctuation marks and the corresponding duration is 7068 milliseconds, then the duration occupied by the punctuation marks is 4 * 10 = 40 milliseconds. The other words are evenly distributed as (7068 - 40) ÷ (30 - 4) = 270. Each ordinary character occupies approximately 270 milliseconds. Based on the duration of ordinary characters, the duration occupied by punctuation marks, and the start - end time points of each segment of the spoken text in the spoken text segmentation data group, an array of single - word durations can be obtained, which can be expressed as [{"start": 0,"end": 270,"content": "As"},{"start": 270,"end": 540,"content": "educators"},{"start": 540,"end": 810,"content": "we"},{"start": 810,"end": 1081,"content": "not"},{"start": 1081,"end": 1351,"content": "only"},{"start": 1351,"end": 1361,"content": "are"},{"start": 1361,"end": 1631,"content": "but"},{"start": 1631,"end": 1902,"content": "also"},{"start": 1902,"end": 2172,"content": "…"}].

[0063] Subsequently, a three-character duration array is generated from the single-character duration array. This is because when matching sentences, if single characters are used for collision matching, the occurrence of the same character multiple times in a sentence will cause timing errors. Using single characters for subtitle alignment will result in a relatively large error. Therefore, it is necessary to assemble characters into three-character groups for collision matching. The principle is that each single character is merged with two characters after it, with the in point using the in point of the current character and the out point using the out point of the third character. The phrase "as an educator" will generate the following array: "as a teacher", "for education", "educator", etc., looping until the end, resulting in a three-character duration array of the spoken text, which can be represented as: [{"start": 0,"end": 810,"content": "as a teacher"},{"start": 270,"end": 1081,"content": "for education"},{"start": 540,"end": 1351,"content": "educator"},{"start": 810,"end": 1361,"content": "educator,"},{"start":1081,"end": 1631,"content": "educator, I"},{"start": 1351,"end": 1902,"content": ", we"},{"start": 1361,"end": 2172,"content": "we do not"},{"start": 1631,"end":2442,"content": "They are not only"},{"start": 1902,"end": 2713,"content": "not only"},{"start": 2172,"end": 2983,"content": "only"},…).

[0064] Subsequently, the start-end times of each subtitle in the subtitle set are corrected to obtain standard subtitle data. Specifically, each subtitle in the subtitle set is traversed. For each subtitle, three characters are taken and compared with three-character groups in the three-character duration array. If a three-character group with the same three characters exists, it is added to the hit set list, and the position index of the collided three-character group in the three-character duration array is recorded. In the next collision query, the query starts from the recorded index to avoid duplicate collisions.

[0065] After collision detection, it is judged whether the length of the set is greater than 0. If it is greater than 0, it proves that there is a collision point, and the entry point of this collision and the exit point of the last collision are recorded; if it is equal to 0, it proves that there is no collision point, and the entry and exit points are not defined. Then, divide the number of successful collisions by the total length of the current item text to obtain the hit rate of the current item. The hit rate ranges from 0 to 1, where 0 means no hit at all and 1 means all hits.

[0066] Determine the hit rates of each item in the subtitle set. When the hit rate is greater than a certain threshold (here it is taken as 0.5), the entry and exit points of this item are corrected; otherwise, it remains unchanged. When correcting, first determine whether it is a complete hit. If it is a complete hit, modify the entry point of the current item to the recorded collision entry point, and then look up one item forward. If it is found that the entry point of the current item is less than the exit point of the previous item, it means there is an intersection, and the exit point of the previous item needs to be updated to the entry point of the current item, and then judge forward until there is no intersection. Finally, modify the exit point of the current item to the recorded collision exit point to complete the correction.

[0067] If it is not a complete hit, based on the character distance between the first character of the first hit character group to be matched and the first character in the subtitle, modify the entry point time of the first three-character group matched in the subtitle as the initial entry point time; based on the character distance between the last character of the last hit character group to be matched and the last character in the subtitle, modify the exit point time of the last three-character group matched in the subtitle as the corrected exit point time.

[0068] For example, in one case, the hit rate is greater than 50%, but it does not start with the first character or end with the last character. For example, "as a teacher", "for education", "educator", only the time points (270 - 1081) of the three characters "for education" are found, but 3 characters are collided, and the collision rate is 60%. When calculating the collision entry point here, it is calculated that there is still one character in front that is not collided, and 270 - (single character duration) is used to get the final collision entry point. The calculation of the collision exit point is the same. It is calculated that there is still one character behind that is not collided, and 1081 + (single character duration) is used to get the final collision exit point. Finally, organize and obtain the final corrected data, and剔除 the collision data and other irrelevant information.

[0069] Step S60, generate subtitles in the corresponding oral broadcast file based on the standard subtitle data.

[0070] After obtaining the final corrected subtitles, while playing the oral broadcast audio, display the corresponding subtitles according to the progress of the oral broadcast audio to complete the final rendering effect.

[0071] Understandably, this embodiment generates audio segments from spoken text using the actual segmentation method of the speech generation model and obtains accurate start-end times. It then combines this with the text length of the subtitles themselves for initial timeline allocation. Finally, it utilizes a three-character group matching and hit rate correction mechanism to precisely correct the initial entry and exit points of each subtitle, achieving a leap from coarse-grained average allocation to fine-grained character-level time positioning. This method effectively solves problems in traditional subtitle alignment caused by uncertain model pauses, inconsistent text segmentation rules, and time misalignment due to repeated characters. It significantly improves the synchronization accuracy between subtitles and spoken speech, making the subtitles more natural, smooth, and in line with the speech rhythm, thereby greatly improving the automation level of subtitle production and the viewing experience in generative spoken speech scenarios. For specific effects, please refer to... Figure 2 Using this embodiment, as a comparison with the average distribution of subtitle duration, the duration of almost all subtitles has been corrected.

[0072] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method of aligning spoken subtitles in this application. Any simple variations based on this technical concept are within the protection scope of this application.

[0073] This application also provides a device for aligning spoken subtitles; please refer to [reference needed]. Figure 4 The voice-over subtitle alignment device includes: The audio generation module 10 is used to process the spoken text into segments and input them sequentially into the speech generation model for speech generation processing to obtain the audio data corresponding to each segment of spoken text. The audio splicing module 20 is used to splice the audio data of each segment to obtain the spoken audio and determine the start-end time points of each spoken text segment. The subtitle segmentation module 30 is used to segment the spoken text according to the subtitle segmentation rules to obtain a subtitle set; The subtitle alignment module 40 is used to determine the initial entry and exit points of each subtitle in the subtitle set based on the total duration of the spoken audio and the subtitle length of each subtitle in the subtitle set; The subtitle correction module 50 is used to correct the initial entry and exit points of each subtitle based on the start-end time points of each segment of spoken text, determine the entry and exit points of each subtitle, and obtain standard subtitle data. The subtitle generation module 60 is used to generate subtitles in the corresponding audio file based on standard subtitle data.

[0074] The audio-visual text alignment device provided in this application employs the audio-visual text alignment method in the above embodiments, which can solve the technical problem of low matching accuracy in audio-visual text alignment in related technologies. Compared with related technologies, the beneficial effects of the audio-visual text alignment device provided in this application are the same as those of the audio-visual text alignment method provided in the above embodiments, and other technical features in the audio-visual text alignment device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0075] This application provides a voice-over subtitle alignment device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the voice-over subtitle alignment method in the above embodiments.

[0076] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the spoken caption alignment device in the embodiments of this application. The spoken caption alignment device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The illustrated voice-over caption alignment device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0077] like Figure 4As shown, the voice-over subtitle alignment device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the voice-over subtitle alignment device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the voice-over captioning device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show voice-over captioning devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0078] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0079] The audio-visual text alignment device provided in this application, employing the audio-visual text alignment method in the above embodiments, can solve the technical problem of low matching accuracy in audio-visual text alignment in related technologies. Compared with related technologies, the beneficial effects of the audio-visual text alignment device provided in this application are the same as those of the audio-visual text alignment method provided in the above embodiments, and other technical features in this audio-visual text alignment device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0080] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0081] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0082] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the voice-over subtitle alignment method in the above embodiments.

[0083] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0084] The aforementioned computer-readable storage medium may be included in the audio caption alignment device; or it may exist independently and not assembled in the audio caption alignment device.

[0085] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the audio-visual subtitle alignment device, the audio-visual subtitle alignment device performs the following actions: segments the audio text and sequentially inputs it into a speech generation model for speech generation processing to obtain audio data corresponding to each segment of audio text; concatenates the audio data segments to obtain audio-visual data and determines the start-end time points of each segment of audio text; divides the audio text according to subtitle segmentation rules to obtain a subtitle set; determines the initial entry / exit points of each subtitle in the subtitle set based on the total duration of the audio-visual data and the subtitle length of each subtitle in the subtitle set; corrects the initial entry / exit points of each subtitle based on the start-end time points of each segment of audio text, determines the subtitle entry / exit points of each subtitle, and obtains standard subtitle data; and generates subtitles in the corresponding audio-visual file based on the standard subtitle data.

[0086] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0088] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0089] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described audio subtitle alignment method, thereby solving the technical problem of low matching accuracy in audio subtitle alignment in related technologies. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the audio subtitle alignment method provided in the above embodiments, and will not be repeated here.

[0090] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method for aligning spoken subtitles.

[0091] The computer program product provided in this application can solve the technical problem of low matching accuracy in the alignment of spoken subtitles in related technologies. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the spoken subtitle alignment method provided in the above embodiments, and will not be repeated here.

[0092] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for aligning spoken subtitles, characterized in that, The method includes: The spoken text is segmented and processed, and then sequentially input into the speech generation model for speech generation processing to obtain the audio data corresponding to each segment of spoken text. The audio data segments are spliced ​​together to obtain the spoken audio, and the start-end time points of each spoken text segment are determined. The spoken text is divided according to the subtitle division rules to obtain a subtitle set; Based on the total duration of the spoken audio and the length of each subtitle in the subtitle set, the initial entry and exit points of each subtitle in the subtitle set are determined; Based on the start-end time points of each spoken text segment, the initial entry and exit points of each subtitle are corrected to determine the entry and exit points of each subtitle and obtain standard subtitle data. Based on the standard subtitle data, subtitles are generated in the corresponding audio file.

2. The method for aligning spoken subtitles as described in claim 1, characterized in that, The step of segmenting the spoken text and inputting it sequentially into a speech generation model for speech generation to obtain the audio data corresponding to each segment of spoken text includes: Based on the type of the invoked speech generation model, the segmentation processing method of the spoken text is determined to obtain multiple segments of text to be processed; Each segment of text to be processed is input into the speech generation model in sequence to obtain the corresponding audio data, and the start time of each audio data is recorded. According to the input order of the text to be processed, the audio data are concatenated into a spoken audio.

3. The method for aligning spoken subtitles as described in claim 1, characterized in that, The step of dividing the spoken text according to a preset subtitle division rule to obtain a subtitle set includes: Based on the line breaks in the spoken text, the spoken text is split into its first segmentation, and the first segmentation result is obtained. The first splitting result is further split according to the subtitle design length to obtain the second splitting result; Remove the punctuation marks at the end of each subtitle in the second splitting result to obtain the subtitle set.

4. The method for aligning spoken subtitles as described in claim 3, characterized in that, Based on the total duration of the spoken audio and the length of each subtitle in the subtitle set, the steps for determining the initial entry and exit points of each subtitle in the subtitle set include: Determine the length of each subtitle in the subtitle set and the total length of all subtitles; Iterate through each subtitle in the set and determine the duration ratio of the subtitle based on the ratio of the subtitle length to the total subtitle length; Based on the duration ratio and the total duration of the spoken audio, the initial entry and exit points of each subtitle in the subtitle set are determined.

5. The method for aligning spoken subtitles as described in claim 4, characterized in that, The steps of correcting the initial entry and exit points of each subtitle based on the start-end time points of each spoken text segment, determining the entry and exit points of each subtitle, and obtaining standard subtitle data include: Based on the start-end time points of the spoken text and the text length of each segment of spoken text, the start-end time of each character in each segment of spoken text is determined to obtain a single-character duration array. For each character in the single-character duration array, take one character forward and one character backward to form a three-character group, determine the start-end time point of each three-character group, and generate a three-character duration array; For each subtitle in the subtitle set, a sampling window size of three characters and a sampling sliding step of one character are used to sequentially extract multiple groups of characters to be matched from the subtitle; wherein the last character in the last group of characters to be matched is the same as the last character in the corresponding subtitle. For each character group to be matched in any subtitle, find a three-character group that matches completely from the three-character duration array, and count the number of character groups to be matched that match the three-character group in each subtitle as the number of collisions; For each subtitle in the subtitle set, the hit rate of the subtitle is determined based on the ratio of the number of collisions to the subtitle length; Subtitles that meet the hit rate requirements are corrected to obtain the standard subtitle data.

6. The method for aligning spoken subtitles as described in claim 5, characterized in that, The step of correcting the subtitles that meet the hit rate requirements to obtain the standard subtitle data includes: For each subtitle in the subtitle set, if the subtitle is a perfect match, the in-time of the subtitle is modified to the in-time of the first matched three-character group in the subtitle, as the initial in-time; the out-time of the subtitle is modified to the out-time of the last matched three-character group in the subtitle, as the corrected out-time. If the subtitle is not a complete match, the in-time of the first three-character group matched by the subtitle is modified based on the character distance between the first character of the first matched character group and the first character of the subtitle, and is used as the initial in-time; the out-time of the last matched character group is modified based on the character distance between the last character of the last matched character group and the last character of the subtitle, and is used as the corrected out-time. If the initial in-point time is less than the out-point time of the previous subtitle in the subtitle set, then the initial in-point time is modified to the out-point time of the previous subtitle to obtain the corrected in-point time; Standard subtitle data is determined based on the correction in time and the correction out time.

7. A device for aligning spoken subtitles, characterized in that, The device includes: The audio generation module is used to process the spoken text into segments and input them sequentially into the speech generation model for speech generation processing, so as to obtain the audio data corresponding to each segment of spoken text. The audio splicing module is used to splice together audio data from different segments to obtain spoken audio and determine the start and end time points of each spoken text segment. The subtitle segmentation module is used to segment the spoken text according to subtitle segmentation rules to obtain a subtitle set; The subtitle alignment module is used to determine the initial entry and exit points of each subtitle in the subtitle set based on the total duration of the spoken audio and the subtitle length of each subtitle in the subtitle set. The subtitle correction module is used to correct the initial entry and exit points of each subtitle based on the start-end time points of each spoken text segment, determine the subtitle entry and exit points of each subtitle, and obtain standard subtitle data. The subtitle generation module is used to generate subtitles in the corresponding audio file based on the standard subtitle data.

8. A voice-over subtitle alignment device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the spoken subtitle alignment method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the audio-visual subtitle alignment method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the spoken subtitle alignment method as described in any one of claims 1 to 6.

Images