Efficient subtitle alignment method for mandarin video

Abstract

The invention discloses an efficient subtitle alignment method for mandarin videos. The method comprises the following steps: grouping videos and subtitle texts according to video durations, and marking a starting time point and an ending time point of each group; manually marking the starting time and the ending time of each group of video effective subtitles; extracting audios from the video, segmenting the audios according to groups, and transferring the segmented audios into text information and time information through a speech recognition model; converting the caption text and the character information into pinyin by using a dictionary data set, and constructing an ordered array set according to a single character sequence; matching the array set of each group of subtitle texts with the array set of each group of character information, and performing alignment according to the starting time and the ending time of the effective subtitles to obtain an alignment result; and combining all the groups according to the start time point and the end time point of each group based on the alignment result to obtain aligned subtitles. The corresponding relation between the voice and the text is automatically detected, and the manual workload is greatly reduced.

Description

Technical Field

[0001] The invention belongs to the technical field of subtitle alignment, and in particular relates to an efficient subtitle alignment method for Mandarin videos. Background Art

[0002] There are usually two ways to align subtitles. One is to align the transcribed content and timeline information through manual dictation, where videos are divided into those with hard subtitles and those without subtitles. The advantage of this solution is that the alignment accuracy is relatively higher, but the disadvantage is that it takes a very long time. The other is to use hard subtitle recognition and transcription technology to send the input video file to the hard subtitle generation engine, obtain the transcription output results and align them. The disadvantage is that it cannot be applied to audio and videos without hard subtitles, and the use of OCR (optical character recognition) subtitle recognition technology takes longer and has slightly lower energy efficiency. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an efficient subtitle alignment method for Mandarin videos to solve the above problems in the prior art.

[0004] To achieve the above object, the present invention provides an efficient subtitle alignment method for Mandarin videos, comprising the following steps:

[0005] Group videos and subtitle texts according to video length and mark the start and end time points of each group;

[0006] Manually mark the start and end time of each set of video valid subtitles;

[0007] Extract audio from the video and segment it into groups, and transcribe the segmented audio into text information and time information through the speech recognition model;

[0008] Convert subtitle text and text information into pinyin using a dictionary data set and construct an ordered array set according to the order of the individual words;

[0009] Match the array set of each group of subtitle texts with the array set of each group of text information, and align them according to the start time and end time of the valid subtitles to obtain an alignment result;

[0010] Based on the alignment result, all the groups are merged according to the start time point and the end time point of each group to obtain aligned subtitles.

[0011] Preferably, the method for grouping videos and subtitle texts according to video duration includes: setting a segmentation duration threshold, grouping according to the segmentation duration threshold, and simultaneously making the subtitle text grouping correspond to the video grouping to obtain corresponding groups of video and text.

[0012] Preferably, the expressions for the start time point and the end time point of each group are:

[0013] The starting time of each group = effective subtitle start time + 20*60*(i-1);

[0014] The end time of each group = effective subtitle start time + 20*60*i;

[0015] Where i is the group number, i>=1.

[0016] Preferably, the speech recognition model includes a whisper speech recognition model.

[0017] Preferably, the method for obtaining the alignment result comprises:

[0018] Calculating the similarity ratio between the subtitle text and the pinyin word sequence of the text information, and performing subtitle text matching and adding according to the similarity ratio;

[0019] The added subtitle text is verified, and after the verification is passed, the subtitle text is checked, and the unmatched subtitle text is inserted into the middle position of the upper and lower sentences according to the upper and lower sentence information.

[0020] Preferably, the method for obtaining the similarity includes: inputting two character strings, using a two-dimensional array to record whether each character is the same, if they are the same, record them as 0, if they are not the same, record them as 1, accumulate the number of identical characters in each row and column, then the last number in the array is the total number of differences, taking the length of the two character strings divided by the total number of differences is the dissimilarity, and subtracting the dissimilarity from 1 is the similarity.

[0021] Preferably, the method for verifying the added subtitle text includes:

[0022] Check whether a subtitle text is completely included in the time range of another subtitle text. If so, the included subtitle text time will be reset to zero and re-matched with other text information.

[0023] Preferably, the alignment result is in SRT format.

[0024] Compared with the prior art, the present invention has the following advantages and technical effects:

[0025] The present invention uses an algorithm to automatically detect the correspondence between speech and text, greatly reducing the workload of manual dictation and timeline alignment, and significantly improving the speed and scale of subtitle production, especially when processing long videos or series. The present invention is not limited to videos with hard subtitles, but can also process videos without subtitles and pure audio files, expanding the application scope of the technology and improving the flexibility and versatility of subtitle production.

[0026] The present invention reduces the reliance on human resources and reduces the labor cost in the process of subtitle production. It improves the accuracy and speed of subtitle production, reduces the extra correction work caused by insufficient OCR technology, and thus saves time and economic costs. The improvement of cost-effectiveness makes subtitle services more popular, especially in the fields of education, entertainment and barrier-free, which helps more people enjoy audio and video content without barriers and promotes social inclusiveness and diversity.

[0027] The high-precision alignment of the invention ensures synchronization of subtitles with audio content, ensuring accurate communication of meaning even in the case of translated subtitles, improving the user's viewing experience. For subtitles that are blurred, overlapping, or in non-standard fonts, the new technology can provide higher recognition accuracy, reducing errors caused by dialects, accents, lack of concentration due to long hours of work, or differences in listening comprehension. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is an overall flow chart of the method of an embodiment of the present invention;

[0030] Figure 2 FIG. 4 is a flow chart of an alignment process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Embodiment 1

[0034] like Figure 1 As shown, this embodiment provides an efficient subtitle alignment method for Mandarin videos, comprising the following steps:

[0035] Group videos and subtitle texts according to video length and mark the start and end time points of each group;

[0036] Manually mark the start and end time of each set of video valid subtitles;

[0037] Extract audio from the video and segment it into groups, and transcribe the segmented audio into text information and time information through the speech recognition model;

[0038] Convert subtitle text and text information into pinyin using a dictionary data set and construct an ordered array set according to the order of the individual words;

[0039] Match the array set of each group of subtitle texts with the array set of each group of text information, and align them according to the start time and end time of the valid subtitles to obtain an alignment result;

[0040] Based on the alignment result, all the groups are merged according to the start time point and the end time point of each group to obtain aligned subtitles.

[0041] The specific implementation method is as follows:

[0042] 1. Upload audio, video and subtitle text, and group them into groups of 20 minutes according to the length of the video. At the same time, adjust the alignment of the text grouping to the video grouping, and finally obtain the corresponding minutes of the video and text and the time of each group corresponding to the video.

[0043] 2. When manually adjusting the playback progress of the loaded video, you can mark the start time and end time of the valid subtitles of the video respectively. This operation records the effective alignment time range of the video.

[0044] 3. Based on the information obtained in steps 1 and 2, calculate the audio that needs to be extracted from the original video and cut it. The cutting information is the effective start time, effective end time, and start and end time of each group of the original video.

[0045] The calculation formula is:

[0046] The start time of each group = effective start time + 20*60*(i-1)

[0047] End time of each group = effective start time + 20*60*i

[0048] Where i is the group number, i>=1. The time unit for calculation is seconds. If it is the last group in the group, the end time is the effective end time.

[0049] The FFmpeg audio extraction command is as follows:

[0050] ffmpeg-iinput.mp4-vn-acodec copy output.mp3

[0051] The functions of each parameter are as follows:

[0052] -iinput.mp4: specifies the input video file path.

[0053] -vn: indicates that the video stream is not included and only the audio is retained.

[0054] -acodec copy: Indicates that the audio codec is set to copy mode, which means that if the output format supports the codec of the source audio, it will directly copy the audio stream without any conversion, which can ensure audio quality and is very fast.

[0055] output.mp3: specifies the output audio file name and format.

[0056] The FFmpeg cutting command is as follows:

[0057] ffmpeg-iinput.mp3-ss 00:00:00-t 00:20:00-c copy output.mp3

[0058] The functions of each parameter are as follows:

[0059] -iinput.mp4 specifies the input video file.

[0060] -ss 00:00:00 sets the start time, this is the beginning of the video.

[0061] -t 00:20:00 sets the duration, that is, the length of the interception, which is 20 minutes here.

[0062] -c copy This means that the encoder will be set to "copy" mode. If the source and destination files are in the same format, this will be a lossless copy process, which can save a lot of processing time and resources. However, if the destination file format is different from the source file, ffmpeg will re-encode it.

[0063] 4. Use an ASR (speech recognition) model, such as the open source whisper speech recognition model or other speech recognition models, to transcribe the extracted audio of each group into text and time information.

[0064] 5. Preprocess Mandarin data, convert the corresponding text and transcribed text of each group into pinyin using the dictionary data set and form an ordered array set according to the single word. Since there are a large number of polyphonic characters in Chinese, especially the recognition of names, for example, the original text is Zhang Xiaoli, and the phonetic transcription is likely to be Zhang Xiaoli. This polyphonic situation will lead to misjudgment of text alignment matching, and phonetic transcription is essentially the description of the phonetic transcription of text through phonetic transcription. Therefore, in order to solve the mismatch caused by Chinese polyphonic characters, the text is converted into pinyin words, such as Zhang Xiaoli is converted into zhang xiao li, and Zhang Xiaoli is also converted into zhang xiao li. In this way, even if the text of the phonetic transcription is different but the pronunciation is similar, this method can still match it.

[0065] 6. Align text (such as Figure 2 shown):

[0066] 6.1 Text matching: For each text that needs to be aligned, traverse all the transcribed texts to find matching pinyin word sequences.

[0067] Compute the similarity ratio between the phonetic representation of the aligned text and the word sequence in the transcribed text.

[0068] If the similarity exceeds a certain threshold (SIMILARY), for example: 0.7. and the time and content do not match any existing subtitles in the subtitles, then set the start and end time of the current subtitle.

[0069] Similarity algorithm: Input two strings and use a two-dimensional array to record whether each character is the same. If they are the same, record them as 0, and if they are different, record them as 1. The number of identical characters in each row and column is accumulated, and the last number in the array is the total number of differences. The length of the two strings divided by the total number of differences is the dissimilarity, and 1 minus the dissimilarity is the similarity.

[0070] 6.2 Correction and verification: After all subtitles are added, the code checks whether there are subtitles that are completely surrounded by another subtitle time range, that is, their start time is greater than the start time of a subtitle and their end time is less than the end time of a subtitle. If found, their times will be reset to zero, indicating that realignment is needed.

[0071] 6.3 The subtitles that need to be realigned will go through a matching process similar to the initial process again, that is, ignoring the time periods that have been matched, and setting the start and end time of the current subtitles if the matching similarity in the free time periods is greater than SIMILARY.

[0072] 6.4 Additional Check and Final Processing: Further process the subtitles to ensure that there are no unmatched subtitles. For the unmatched text, place it in the middle of the upper and lower sentences according to the upper and lower sentence information.

[0073] 6.5 Output results: The output results are in SRT format.

[0074] 7. Merge the results of each group: Merge all groups into one result based on the retained start time of each group.

[0075] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An efficient subtitle alignment method for Mandarin videos, characterized in that: The following steps are involved: Group videos and subtitle texts according to video length and mark the start and end time points of each group; Manually mark the start and end time of each set of video valid subtitles; Extract audio from the video and segment it into groups, and transcribe the segmented audio into text information and time information through the speech recognition model; Convert subtitle text and text information into pinyin using a dictionary data set and construct an ordered array set according to the order of the individual words; Match the array set of each group of subtitle texts with the array set of each group of text information, and align them according to the start time and end time of the valid subtitles to obtain an alignment result; Based on the alignment result, all the groups are merged according to the start time point and the end time point of each group to obtain aligned subtitles.

2. The efficient subtitle alignment method for Mandarin videos according to claim 1, characterized in that: The method for grouping videos and subtitle texts according to video duration includes: setting a segmentation duration threshold, grouping according to the segmentation duration threshold, and simultaneously making the subtitle text grouping correspond to the video grouping to obtain corresponding groups of video and text.

3. The efficient subtitle alignment method for Mandarin videos according to claim 1, characterized in that: The expressions for the start and end time points of each group are: The starting time of each group = effective subtitle start time + 20*60*(i-1); The end time of each group = effective subtitle start time + 20*60*i; Where i is the group number, i>=1.

4. The efficient subtitle alignment method for Mandarin videos according to claim 1, characterized in that: The speech recognition model includes a whisper speech recognition model.

5. The efficient subtitle alignment method for Mandarin videos according to claim 1, characterized in that: The method for obtaining the alignment result comprises: Calculating the similarity ratio between the subtitle text and the pinyin word sequence of the text information, and performing subtitle text matching and adding according to the similarity ratio; The added subtitle text is verified, and after the verification is passed, the subtitle text is checked, and the unmatched subtitle text is inserted into the middle position of the upper and lower sentences according to the upper and lower sentence information.

6. The efficient subtitle alignment method for Mandarin videos according to claim 5, characterized in that: The method for obtaining the similarity comprises: inputting two character strings, using a two-dimensional array to record whether each character is the same, if the same is recorded as 0, if not the same is recorded as 1, accumulating the number of the same characters in each row and column, then the last number in the array is the total number of differences, taking the length of the two character strings divided by the total number of differences is the dissimilarity, and subtracting the dissimilarity from 1 is the similarity.

7. The efficient subtitle alignment method for Mandarin videos according to claim 5, characterized in that: Methods for verifying the added subtitle text include: Check whether a subtitle text is completely included in the time range of another subtitle text. If so, the included subtitle text time will be reset to zero and re-matched with other text information.

8. The efficient subtitle alignment method for Mandarin videos according to claim 5, characterized in that: The alignment result is in SRT format.

Images