Subtitle generation model design method and device, storage medium and program product

Abstract

The invention provides a subtitle generation model design method and device, a storage medium and a program product, and relates to the technical field of natural language processing. The subtitle generation model design method comprises the following steps: inputting a section of voice, corresponding correct content and the maximum limited number of words in a single row of subtitles; carrying out human voice detection by using a human voice detection model based on deep learning; performing voice recognition by using a pre-trained depth model; matching and correcting the correct content with the recognized characters to obtain subtitle characters; and according to the maximum limited word number of the single row of the subtitles, subtitle characters are divided and integrated, and a final subtitle result is obtained. According to the subtitles generated by the method, actual voice pause and text language habits are comprehensively considered, the subtitles and the audio are higher in matching degree, and subtitle characters are easier to read; according to the method, the maximum limited word number of the single row of the subtitles limited by equipment or artificially specified is considered at the same time, and the readability of the generated subtitles is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a subtitle generation model design method, device, storage medium and program product. Background Art

[0002] Subtitle production plays an important role in film and television production. Good subtitles have high requirements for text accuracy and readability, that is, the content of the subtitles must be error-free and the time of subtitle display must correspond to the audio content.

[0003] The prior art applies speech recognition to subtitle production to automatically convert audio content into corresponding text, but the automatically recognized content often cannot meet the accuracy requirements of subtitles. Therefore, the prior art outputs accurate subtitles by giving correct content and speech.

[0004] However, the subtitles generated by the above-mentioned prior art still have problems with readability. That is, in actual applications, the number of words displayed in a single line of subtitles is often limited. Under the word limit, directly outputting subtitles according to the audio sequence without content arrangement often results in subtitles that do not conform to language habits, affecting the readability of the subtitles. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a subtitle generation method, device, storage medium and program product. In the process of generating subtitles, human voice detection and word segmentation technology are fully utilized. The generated subtitles comprehensively consider the actual voice pauses and text language habits, so that the generated subtitles have a higher match with the audio, and the subtitle text is easier to read; the present invention also takes into account the maximum number of characters in a single line of subtitles limited by equipment or artificially stipulated, and further improves the readability of the generated subtitles.

[0006] The meanings of the relevant terms used in the present invention are as follows: subtitle results represent the final results obtained by the present invention; subtitle results are composed of subtitle text and corresponding timestamps; timestamps represent the corresponding start time and end time of an object in the original audio, and this object can be a subtitle word or a segmented speech.

[0007] In a first aspect, the present invention provides a subtitle generation model design method, comprising the following steps:

[0008] Enter a voice and the corresponding correct content, as well as the maximum number of characters in a single line of subtitles;

[0009] Use a deep learning-based voice detection model to detect the input voice and obtain the initial voice segment and corresponding timestamp;

[0010] Post-process the initial segment to obtain segmented speech and corresponding timestamps;

[0011] Input the segmented speech into the pre-trained deep model for speech recognition, and obtain the recognized text and the timestamp corresponding to each text;

[0012] Use the correct content to match and correct the recognized text to obtain subtitle text;

[0013] According to the maximum number of characters in a single line of subtitles, the subtitle text is divided into lines and integrated to obtain the final subtitle result.

[0014] As a further improvement of the present invention, the maximum number of characters in a single line of subtitles is a number of characters determined according to a display device or user requirements.

[0015] As a further improvement of the present invention, the method of using a deep learning-based voice detection model to perform voice detection on the input voice to obtain an initial voice segment and a corresponding timestamp includes:

[0016] Cut the input speech into segments of preset size and input them into the human voice detection model;

[0017] Perform voice discrimination based on the output result of the voice detection model;

[0018] The segments with continuous human voice are merged to obtain the initial speech segments and corresponding timestamps.

[0019] As a further improvement of the present invention, the post-processing of the initial segment to obtain segmented speech and corresponding timestamps includes:

[0020] Set thresholds;

[0021] The threshold is used for discrimination and processing, and the segments that are too short are deleted, and the segments with shorter time intervals are merged.

[0022] As a further improvement of the present invention, the subtitle text is divided into lines and integrated according to the maximum number of characters in a single line of subtitles, including:

[0023] Calculate the number of subtitle characters corresponding to each segmented speech;

[0024] If the number of words does not exceed the maximum limit of a single line of subtitles, the timestamp of the segmented speech and the corresponding subtitle text are directly used to generate a single-line subtitle result;

[0025] If the subtitle text corresponding to a certain segment of speech exceeds the maximum number of characters in a single line of subtitles, the subtitle results will be generated through line division and integration operations.

[0026] As a further improvement of the present invention, the branching and integration operations include:

[0027] Set up sub-segments for segmented speech that needs line breaks;

[0028] Segment the subtitle text corresponding to the segmented speech to obtain a segmentation list and the timestamp of each word;

[0029] Put the words in the word list into sub-segments in order, and control the length of the sub-segment to be no greater than the maximum number of words in a single line of subtitles;

[0030] Integrate the sub-segments and corresponding timestamps to obtain a single-line subtitle result;

[0031] Integrate the single-line subtitle results to get the final subtitle result.

[0032] As a further improvement of the present invention, the integrating sub-segments and corresponding timestamps includes acquiring the timestamp of the sub-segment in the original audio according to the timestamp of each word in the sub-segment.

[0033] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0034] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0035] In a fourth aspect, the present invention provides a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0036] Compared with the prior art, the present invention proposes a subtitle generation method, device, storage medium and program product. In the process of generating subtitles, human voice detection and word segmentation technology are fully utilized. The generated subtitles comprehensively consider the actual voice pauses and text language habits, so that the generated subtitles have a higher match with the audio, and the subtitle text is easier to read; the present invention also takes into account the maximum number of characters in a single line of subtitles limited by equipment or artificially stipulated, and further improves the readability of the generated subtitles. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention discloses a flowchart of a method for designing a subtitle generation model. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models, simulation environments, and software described in the present invention are not the only limiting methods of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0041] In a first aspect, the present invention provides an embodiment of a subtitle generation model design method, such as Figure 1 As shown, the specific process can be as follows:

[0042] S1: Input a voice and the corresponding correct content, as well as the maximum number of characters in a single line of subtitles.

[0043] S2: Use the deep learning-based voice detection model to detect the input voice and obtain the initial voice segment and the corresponding timestamp. Specifically, the following steps are included:

[0044] S21: Cut the input speech into small segments and input them into a human voice detection model, where the human voice detection model uses a deep learning model.

[0045] Preferably, the fragment size is set in advance.

[0046] In one embodiment of the present invention, the segment size is consistent with the data for training the human voice detection model, and is 32ms (milliseconds), 64ms, etc.

[0047] S22: Perform voice identification based on the output result of the voice detection model.

[0048] Preferably, the human voice detection model scores the input speech segment, and the closer the score is to 1, the greater the probability of being a human voice. On this basis, human voice discrimination is performed by setting a discrimination threshold, that is, when the score output by the model is greater than the discrimination threshold, it is considered that there is a human voice in the segment.

[0049] In one embodiment of the present invention, the discrimination threshold is set to 0.5 as a hyperparameter of the human voice detection model.

[0050] S23: Merge the segments containing continuous human voice to obtain an initial voice segment containing human voice and a corresponding time stamp.

[0051] S3: Post-process the initial segment to obtain segmented speech and the timestamp of each segmented speech.

[0052] The post-processing steps are:

[0053] S31: setting a threshold;

[0054] S32: using the threshold in S31 to perform discrimination and processing, deleting segments that are too short, and merging segments with shorter time intervals.

[0055] S4: Input the segmented speech into the pre-trained deep model for speech recognition to obtain the recognized text and the timestamp corresponding to each text.

[0056] S5: Correct the recognized text using the correct content to obtain subtitle text.

[0057] Match the recognized text with the correct content and perform content correction to obtain the subtitle text.

[0058] The main purpose of this step is to correct the homophonetic errors in the recognition content into correct ones.

[0059] S6: Divide and merge the subtitle text to obtain the final subtitle result.

[0060] Calculate the number of subtitle characters corresponding to each segment of speech.

[0061] If the number of subtitle characters does not exceed the maximum limit of a single line of subtitles, the timestamp of the segmented speech and the corresponding subtitle characters are directly used to generate a single-line subtitle result.

[0062] If the number of subtitle characters exceeds the maximum limit of a single line of subtitles, a sub-segment is set for the segmented speech, and the subtitles are generated through word segmentation, cyclic word extraction, and calculation integration. The specific steps are:

[0063] S61: Word segmentation: Segment the subtitle text corresponding to the segmented speech to obtain a word segmentation list and a timestamp for each word.

[0064] S62: Loop word extraction. The first word in the word segmentation list is extracted and placed in the sub-segment. The length of the sub-segment is calculated. If it does not exceed the maximum number of characters in a single line of subtitles, the second word in the word segmentation table is placed in the sub-segment in sequence. This cycle continues until the length of the sub-segment exceeds the maximum number of characters in a single line of subtitles, or the word segmentation table is empty. If the length of the sub-segment exceeds the maximum number of characters in a single line of subtitles, the last word placed in the sub-segment is placed back in the word segmentation list.

[0065] S63: Calculate integration. According to the timestamp of each word in the sub-segment, obtain the timestamp of the sub-segment in the original audio. Generate a corresponding single-line subtitle result using the sub-segment content and timestamp.

[0066] S64: Continue to execute steps S61 to S63 for the remaining contents of the word segmentation list until the word segmentation list is empty.

[0067] Integrate the obtained single-line subtitle results to obtain the final subtitle result.

[0068] In a second aspect, the present invention provides an embodiment of a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0069] In a third aspect, the present invention provides an embodiment of a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0070] In a fourth aspect, the present invention provides an embodiment of a computer program product, which implements the steps of the method described in the first aspect when the computer program is executed by a processor.

[0071] The present invention proposes a subtitle generation method, device, storage medium and program product. In the process of generating subtitles, human voice detection and word segmentation technology are fully utilized. The generated subtitles comprehensively consider actual voice pauses and text language habits, so that the generated subtitles have a higher matching degree with the audio, and the subtitle text is easier to read; the present invention also takes into account the maximum number of characters in a single line of subtitles limited by equipment or artificially stipulated, and further improves the readability of the generated subtitles.

Claims

1. A subtitle generation model design method, characterized in that: The following steps are involved: Enter a voice and the corresponding correct content, as well as the maximum number of characters in a single line of subtitles; Use a deep learning-based voice detection model to detect the input voice and obtain the initial voice segment and corresponding timestamp; Post-process the initial segment to obtain segmented speech and corresponding timestamps; Input the segmented speech into the pre-trained deep model for speech recognition, and obtain the recognized text and the timestamp corresponding to each text; Use the correct content to match and correct the recognized text to obtain subtitle text; According to the maximum number of characters in a single line of subtitles, the subtitle text is divided into lines and integrated to obtain the final subtitle result.

2. The method according to claim 1, characterized in that The maximum number of characters in a single line of subtitles is a number of characters determined according to the display device or user requirements.

3. The method according to claim 1, characterized in that The method uses a deep learning-based voice detection model to perform voice detection on the input voice to obtain an initial voice segment and a corresponding timestamp, including: Cut the input speech into segments of preset size and input them into the human voice detection model; Perform voice discrimination based on the output result of the voice detection model; The segments with continuous human voice are merged to obtain the initial speech segments and corresponding timestamps.

4. The method according to claim 1, characterized in that: The post-processing of the initial segment to obtain segmented speech and corresponding timestamps includes: Set thresholds; The threshold is used for discrimination and processing, and the segments that are too short are deleted, and the segments with shorter time intervals are merged.

5. The method according to claim 1, characterized in that The subtitle text is divided into lines and integrated according to the maximum limit of the number of words in a single line of subtitles, including: Calculate the number of subtitle characters corresponding to each segmented speech; If the number of words does not exceed the maximum limit of a single line of subtitles, the timestamp of the segmented speech and the corresponding subtitle text are directly used to generate a single-line subtitle result; If the subtitle text corresponding to a certain segment of speech exceeds the maximum number of characters in a single line of subtitles, the subtitle results will be generated through line division and integration operations.

6. The method according to claim 5, characterized in that The branch and integration operations include: Set up sub-segments for segmented speech that needs line breaks; Segment the subtitle text corresponding to the segmented speech to obtain a segmentation list and the timestamp of each word; Put the words in the word list into sub-segments in order, and control the length of the sub-segment to be no greater than the maximum number of words in a single line of subtitles; Integrate the sub-segments and corresponding timestamps to obtain a single-line subtitle result; Integrate the single-line subtitle results to get the final subtitle result.

7. The method according to claim 6, characterized in that The integrating sub-segments and corresponding timestamps includes obtaining the timestamp of the sub-segment in the original audio according to the timestamp of each word in the sub-segment.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images