Subtitle generation method and device, electronic equipment, storage medium and computer product

Abstract

The invention relates to the technical field of computers, and provides a subtitle generation method and device, electronic equipment, a storage medium and a computer product, and the method comprises the steps: carrying out the voice recognition processing of video data in to-be-processed data, and obtaining a text sequence and the timestamp information of each character string in each recognition text segment contained in the text sequence in the video data; performing text correction on each recognized text segment in the text sequence to obtain a corresponding corrected text; performing character alignment on each recognition text segment in the text sequence and the corresponding correction text to obtain an alignment result corresponding to each recognition text segment; according to the alignment result corresponding to each identification text segment and the timestamp information of each character string in the video data, respectively determining the timestamp information of a correction text corresponding to the corresponding identification text segment; and generating a subtitle file based on the corrected text and the timestamp information of the corrected text corresponding to each recognition text segment. The subtitle generation processing speed can be increased, and the display accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a subtitle generation method, device, electronic device, storage medium, and computer product. Background Art

[0002] Traditionally, adding subtitles to videos requires a subtitler to analyze the audio frame by frame, mark the time each text or segment appears, and then add the corresponding text to the video frame at the corresponding time. With the development of artificial intelligence, automatic speech recognition (ASR) technology can now be used to recognize audio and use the recognized text directly in subtitle generation.

[0003] However, manually analyzing audio and then adding subtitles consumes significant human resources and time. Automatic speech recognition technology often generates biased results due to the inherent "hallucination" issues inherent in AI models. This is particularly true for specialized terms (such as molecular structures or drug names), and the subtitles are prone to being out of sync with the audio, requiring further manual correction before use.

[0004] As a result, the current subtitle generation process is slow and / or the display accuracy is insufficient. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a subtitle generation method, device, electronic device, storage medium, and computer product to solve the problems of slow processing speed and / or insufficient display accuracy faced in current subtitle generation.

[0006] The subtitle generation method according to the first embodiment of the present application includes: Performing speech recognition processing on the video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; Performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; Performing text alignment on each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; Determining the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data; A subtitle file is generated based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0007] According to one embodiment of the present application, when text alignment is performed on each recognized text segment in the text sequence with the corresponding corrected text, the following operations are performed for each recognized text segment in the text sequence: The longest common subsequence of the current recognized text segment in the text sequence and the corrected text corresponding to the current recognized text segment are matched to obtain an alignment result corresponding to the current recognized text segment; the alignment result includes the longest common subsequence information of the recognized text segment and the corrected text, and the difference subsequence information between the recognized text segment and the corrected text.

[0008] According to one embodiment of the present application, when performing text correction on each recognized text segment in the text sequence, the following operations are performed on each recognized text segment in the text sequence: Encoding the current recognized text segment in the text sequence to obtain encoded data; Inputting the encoded data into a text correction model to obtain corrected encoded data output by the text correction model; wherein the text correction model is obtained by training a pre-trained large language model based on sample encoded data; the sample encoded data includes an encoding of a sample text and an encoding of a corrected version of the sample text; the sample text is any text containing an erroneous character string; The revised coded data is decoded to obtain a revised text of the current recognized text segment.

[0009] According to one embodiment of the present application, performing speech recognition processing on video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data includes: Performing filtering on the video data in the data to be processed to obtain filtered data; Performing voice activity detection on the filtered data to obtain a detection result; Segmenting the video data according to the detection result to obtain video segment data of corresponding data; Performing automatic speech recognition on each video clip data to obtain a recognition text segment corresponding to each video clip data and timestamp information of each character string in each recognition text segment in the video data; A text sequence of the video data is generated according to each recognized text segment.

[0010] According to one embodiment of the present application, generating a subtitle file based on the corrected text corresponding to each identified text segment and its timestamp information includes: Determine subtitle file format information; The corrected text of each identified text segment and its timestamp information are used to generate a subtitle file according to the subtitle file format information.

[0011] According to one embodiment of the present application, after obtaining the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data, if the data to be processed includes text information generated based on the audio content of the video data, the method further includes: Performing text alignment on the text sequence and the text information to obtain an alignment result of the text sequence; determining the timestamp information of the text information according to the alignment result and the timestamp information of each character string in each recognized text segment of the text sequence in the video data; A subtitle file is generated based on the text information and its time stamp information.

[0012] According to the second embodiment of the present application, a subtitle generation device includes: A speech recognition module is used to perform speech recognition processing on the video data in the data to be processed, and obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; a text correction module, configured to perform text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; A text alignment module is used to align each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; an information determination module, configured to determine, based on an alignment result corresponding to each of the identified text segments and timestamp information of each character string in each of the identified text segments in the video data, timestamp information of a corrected text corresponding to each of the identified text segments; The subtitle generation module is used to generate a subtitle file based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0013] According to an embodiment of the third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-described subtitle generation methods is implemented.

[0014] According to the storage medium of the fourth embodiment of the present application, the storage medium is a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned subtitle generation methods.

[0015] A computer program product according to an embodiment of the fifth aspect of the present application includes a computer program, which, when executed by a processor, implements any of the above-mentioned subtitle generation methods.

[0016] The above one or more technical solutions in the embodiments of the present application have at least the following technical effects: First, speech recognition processing is performed on the video data in the data to be processed to obtain a text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data. Then, text correction is performed on each recognized text segment in the text sequence to obtain the corrected text of the corresponding recognized text segment. This can effectively reduce the impact of the "hallucination" problem and improve the accuracy of the recognition results. Each recognized text segment in the text sequence is further aligned with the corresponding corrected text to obtain the alignment result corresponding to each recognized text segment. Based on the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data, the timestamp information of the corrected text corresponding to the corresponding recognized text segment is determined. Since the text alignment is performed and the timestamp information of the corrected text is determined based on the alignment result, the subtitle text and audio are synchronized when displayed in the subtitle file generated based on the corrected text corresponding to each recognized text segment and its timestamp information. This improves the processing speed of subtitle generation while also improving display accuracy.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flowchart of the subtitle generation method provided in an embodiment of the present application.

[0020] Figure 2 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0021] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0022] In the description of the embodiments of the present application, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0023] In the description of the embodiments of this application, it should be noted that, unless otherwise specified or limited, the terms "connected" and "connection" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium. Those skilled in the art will understand the specific meanings of the above terms in the embodiments of this application based on the specific circumstances.

[0024] In the embodiments of the present application, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, a first feature being "above," "above," and "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0025] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0026] The present application proposes a subtitle generation method, device, electronic device, storage medium and computer product.

[0027] Figure 1 This is a flow chart of the subtitle generation method provided by the embodiment of the present application. Figure 1 As shown, the subtitle generation method includes: Step 110 , performing speech recognition processing on the video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data.

[0028] Step 120 , performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment.

[0029] Step 130 , aligning each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment.

[0030] Step 140 : Determine the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data.

[0031] Step 150: Generate a subtitle file based on the corrected text corresponding to each recognized text segment and its timestamp information.

[0032] It should be noted that the execution entity of the subtitle generation method provided in the embodiment of the present application can be an application for generating subtitles, and the application can be deployed in a computer device, such as a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA).

[0033] It should be noted that the data required for this application are all obtained legally after authorization.

[0034] The application for subtitle generation of the present application can be provided with or connected to a subtitle generation device, thereby controlling the subtitle generation device to execute the subtitle generation method of the present application.

[0035] Specifically, in this application, the user can input a set of data as the data to be processed through the above application. The data to be processed may include video data, and the video data in this application may include audio content.

[0036] For the above-mentioned video data, the present application can segment the video data into multiple video clip data containing short audio content.

[0037] Furthermore, speech recognition processing can be performed on each video segment data separately, and the audio content in each video segment data can be converted into corresponding text. The text obtained for the video segment data is determined as a recognized text segment. Each video segment data can be recognized to obtain one or more text segments. If speech recognition of a video segment data returns text at the word or sentence level, the word or sentence is recorded as a "group" of recognized text segments. If text is returned at the string level, the character is recorded as a "group" of recognized text segments.

[0038] Furthermore, all recognized text segments of each video clip data can be spliced ​​into a text sequence.

[0039] It should be noted that, when performing speech recognition processing, this application can also extract the timestamp information of each string (e.g., a Chinese character or an English word) in the video data. The timestamp information of each string in the video data is the time when the string appears in the audio. For example, if the extracted word "I" appears at the 1st second of the video, the timestamp information of the string in the video data is 1.0 seconds.

[0040] Based on this, the timestamp information of each character string of each recognized text segment in the text sequence is extracted from the video data.

[0041] In this application, after completing speech recognition, timestamp information at the character, word or sentence level can be returned, but no matter what level the timestamp is, it will be mapped to the string level.

[0042] It should be noted that the present application may first collect any text containing an erroneous character string as a sample text, and then encode the sample text through a vocabulary (vocab) to obtain encoded data of the sample text.

[0043] Also, the sample text is corrected manually or by other means, and the text without any string errors after correction is used as the corrected text, and the corrected text is further encoded through a vocabulary to obtain encoded data of the corrected text.

[0044] This application can also use the text information directly as the corrected text when the user enters it for the video data. At the same time, automatic speech recognition technology can be used to perform speech recognition on the video data, and the text obtained by speech recognition can be used as sample text. The corrected text and sample text can then be encoded.

[0045] Furthermore, the encoding data of the sample text and the encoding data of the corrected text may be used together as sample encoding data.

[0046] Furthermore, the pre-trained large language model can be fine-tuned in a supervised manner using sample encoding data, and the fine-tuned large language model can be distilled and quantized to focus on the task of text correction and make the model as small as possible, thereby reducing the computing power and time required for inference.

[0047] The large language model obtained after distillation and quantization is used as the text correction model.

[0048] After obtaining the text correction model, you can deploy it using the large speech model deployment framework to further accelerate the model inference time.

[0049] This application can use its own data to fine-tune, distill, and quantize the generative large language model, so that it only focuses on the text error correction task, reducing the amount of calculation and improving the accuracy of the results.

[0050] Based on this, the text correction model can be used to perform text correction on each recognized text segment in the text sequence, thereby obtaining a corrected text for each recognized text segment.

[0051] In another embodiment, the entire text sequence may be directly input into the text correction model to obtain a corrected text of the entire text sequence.

[0052] Furthermore, the present application can perform text alignment on each recognized text segment in a text sequence with the corresponding corrected text, thereby obtaining an alignment result corresponding to each recognized text segment. For example, the first recognized text segment in a text sequence can be aligned with the corrected text of the recognized text segment (i.e., the corrected text obtained by performing text correction on the recognized text segment) to obtain an alignment result for the recognized text segment.

[0053] The alignment result may include the longest common subsequence information between the identified text segment and the corrected text, and the difference subsequence information between the identified text segment and the corrected text.

[0054] If the corrected text is obtained by correcting the entire text sequence, the text sequence and the corrected text can be directly aligned to obtain an alignment result of the text sequence.

[0055] Furthermore, since the corrected text does not contain timestamp information, for each corrected text, it is necessary to determine the timestamp information of the corrected text based on the alignment result corresponding to the corrected text and the timestamp information of each character string in the corresponding identified text segment in the video data. In this way, each corrected text has timestamp information.

[0056] Specifically, for any corrected text, the longest common subsequence information in the alignment result can be obtained, and the first character string in the longest common subsequence information can be further determined and used as the first common character string between the corrected text and the recognized text segment.

[0057] Furthermore, the timestamp information of the first common string to the last string (non-common string) in the corresponding identified text segment of the revised text can be integrated, and the integrated timestamp information can be used as the approximate (ie, group-level) timestamp information of the revised text.

[0058] If the revised text is obtained by revising the entire text sequence, the timestamp information of the text sequence corresponding to the revised text can be determined in the same manner.

[0059] After obtaining each corrected text and its time stamp information, each identified text segment may be matched with the corrected text and its time stamp information, and a subtitle file may be generated according to the requirements of the subtitle file format.

[0060] If the corrected text is obtained by correcting the entire text sequence, a subtitle file can be generated according to the corrected text of the text sequence and its timestamp information in accordance with the requirements of the subtitle file format.

[0061] Furthermore, after obtaining the subtitle file, this application can use the ASS format, and the front end provides preview and download. Among them, the ASS format is a file format for subtitles, which supports rich styles and special effects and is commonly used in video subtitle production.

[0062] Specifically, the preview result may be a video with subtitles, and the video segment corresponding to each subtitle may be selected.

[0063] The preview result can also be a video with subtitles generated using default images, and you can select the video segment corresponding to each subtitle.

[0064] According to the subtitle generation method of the embodiment of the present application, the video data in the processing data is first subjected to speech recognition processing to obtain the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; then, text correction is performed on each recognized text segment in the text sequence to obtain the corrected text of the corresponding recognized text segment, thereby effectively reducing the impact of the "hallucination" problem and improving the accuracy of the recognition result. Further, each recognized text segment in the text sequence is aligned with the corresponding corrected text to obtain the alignment result corresponding to each recognized text segment; and based on the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data, the timestamp information of the corrected text corresponding to the corresponding recognized text segment is determined respectively; because the text alignment is performed and the timestamp information of the corrected text is determined in combination with the alignment result, the subtitle text and the audio are synchronized when displayed in the subtitle file generated based on the corrected text corresponding to each recognized text segment and its timestamp information. Thus, the processing speed of subtitle generation is improved while the display accuracy is improved.

[0065] In one embodiment, speech recognition processing is performed on video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data, including: Performing filtering on the video data in the data to be processed to obtain filtered data; Performing voice activity detection on the filtered data to obtain a detection result; Segment the video data according to the detection results to obtain video segment data of corresponding data; Automatic speech recognition is performed on each video clip data to obtain the recognition text segment corresponding to each video clip data and the timestamp information of each character string in each recognition text segment in the video data; A text sequence of the video data is generated according to each recognized text segment.

[0066] Specifically, the present application may filter the video data in the data to be processed by a high-pass filter to obtain filtered data (ie, filtered video data).

[0067] Furthermore, voice activity detection (VAD) technology can be used to perform voice activity detection on the filtered data, and labels of corresponding speech segments and silence segments can be obtained as detection results.

[0068] Furthermore, based on the markings of the speech segments and the silence segments in the detection results, the video data can be divided into multiple segments, thereby obtaining multiple video segment data.

[0069] By segmenting the video, the time and computing power consumption for subsequent speech recognition can be reduced.

[0070] Furthermore, for each video segment data, speech recognition can be performed using automatic speech recognition technology, thereby obtaining one or more recognition text segments corresponding to each video segment data.

[0071] The automatic speech recognition technology in this application can also perform word segmentation while recognizing the audio content in the video data. This can be achieved by combining speech recognition and natural language processing technology.

[0072] Furthermore, the present application can extract the timestamp information of each character string in each recognized text segment in the video data.

[0073] Furthermore, each recognized text segment is concatenated and used as a text sequence of the video data.

[0074] The present application performs speech recognition after filtering and segmenting the video data, which can improve the speed and accuracy of speech recognition, thereby obtaining an accurate text sequence, and then performing text correction on each recognized text segment in the text sequence. The obtained corrected text is more accurate, which can effectively reduce the impact of the "hallucination" problem and improve the accuracy and speed of the recognition results.

[0075] In one embodiment, when performing text correction for each recognized text segment in a text sequence, the following operations are performed for each recognized text segment in the text sequence: Encode the current recognized text segment in the text sequence to obtain encoded data; Inputting the encoded data into a text correction model to obtain corrected encoded data output by the text correction model; wherein the text correction model is obtained by training a pre-trained large language model based on the sample encoded data; the sample encoded data includes the encoding of the original text, the encoding of the text corrected by the large language model, and the encoding of the corrected text; The corrected coded data is decoded to obtain the corrected text of the current recognized text segment.

[0076] Specifically, when the text correction model is deployed, the present application can encode each recognized text segment in the text sequence through a vocab to obtain encoded data.

[0077] Furthermore, each piece of coded data is input into the text correction model respectively, and the corrected coded data corresponding to each piece of coded data output by the text correction model can be obtained.

[0078] After obtaining the corrected coded data, the corrected coded data is decoded to obtain the corrected text of the corresponding recognized text segment.

[0079] This application further fine-tunes, distills, and quantizes the pre-trained large language model in a supervised manner, so that the resulting text correction model has stronger text correction capabilities and faster correction processing speed, making the corrected text obtained by performing text correction on each recognized text segment in the text sequence more accurate, thereby effectively reducing the impact of the "hallucination" problem and improving the accuracy and recognition speed of the recognition results.

[0080] In one embodiment, when text alignment is performed on each recognized text segment in the text sequence and the corresponding corrected text, the following operations are performed for each recognized text segment in the text sequence: The longest common subsequence of the current recognized text segment in the text sequence is matched with the corrected text corresponding to the current recognized text segment to obtain the alignment result corresponding to the current recognized text segment; the alignment result includes the longest common subsequence information of the recognized text segment and the corrected text, as well as the difference subsequence information between the recognized text segment and the corrected text.

[0081] Specifically, this application transforms the problem of calculating approximate timestamps at the string level into the longest common subsequence (LCS) problem and uses the diff_match_patch method to match the identified text segment with the corrected text. diff_match_patch is an open source library for processing text difference comparisons (Diff), fuzzy matching (Match), and patch generation (Patch).

[0082] Specifically, when text alignment is performed on each recognized text segment in a text sequence with the corresponding corrected text, for each recognized text segment, the longest common subsequence can be matched with the corresponding corrected text to obtain the alignment result corresponding to the recognized text segment; wherein the alignment result includes the longest common subsequence information of the recognized text segment and the corrected text, as well as the difference subsequence information between the recognized text segment and the corrected text.

[0083] This application performs text alignment on each recognized text segment in a text sequence with the corresponding corrected text, obtaining alignment results corresponding to each recognized text segment. This allows the timestamp information of the corrected text corresponding to each recognized text segment to be determined based on the alignment results corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data. Due to the text alignment and the determination of the timestamp information of the corrected text based on the alignment results, the subtitle text and audio in the subtitle file generated based on the corrected text corresponding to each recognized text segment and its timestamp information are synchronized when displayed. This improves the display accuracy of the generated subtitles.

[0084] This application converts the problem of calculating the text timestamp required for subtitle files into an LCS problem, matches the ASR result text with the source text, and improves the accuracy of the subtitle timestamp.

[0085] In one embodiment, generating a subtitle file based on the corrected text corresponding to each identified text segment and its timestamp information includes: Determine subtitle file format information; The corrected text of each recognized text segment and its timestamp information are used to generate a subtitle file according to the subtitle file format information.

[0086] Specifically, the present application can obtain the required information of the subtitle file format, which may include but is not limited to text format (text size and text color, etc.), timestamp format, timestamp accuracy, timestamp range, etc.

[0087] Therefore, a subtitle file can be generated based on the subtitle file format information, the corrected text of each identified text segment, and its time stamp information.

[0088] If it is necessary to split the corrected text corresponding to the recognized text segment, this can be achieved based on the corrected text and the corresponding waveform information in the video data.

[0089] By performing text alignment and determining the timestamp information of the corrected text based on the alignment results, this application ensures that the subtitle text and audio are synchronized when displayed in the subtitle file generated based on the corrected text corresponding to each identified text segment and its timestamp information. This improves the display accuracy of the generated subtitles.

[0090] In one embodiment, after obtaining the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data, if the data to be processed includes text information generated based on the audio content of the video data, the following steps are further included: Performing text alignment on the text sequence and the text information to obtain an alignment result of the text sequence; determining the timestamp information of the text information according to the alignment result and the timestamp information of each character string in each recognized text segment of the text sequence in the video data; A subtitle file is generated based on the text information and its time stamp information.

[0091] It should be noted that in this application, before the text correction model is trained or deployed, the user may be required to input text information generated based on the content of the video data in addition to the data to be processed. For example, if the video data includes audio content, the user can edit the corresponding text based on the audio content to obtain the text information of the video data.

[0092] After the text correction model is deployed, text information does not need to be entered when providing data to be processed.

[0093] In this application, character-level processing can be performed on text information, including but not limited to punctuation and special character removal, character set mapping conversion and other operations, wherein punctuation and special characters in the input text information are removed, but the position information of the punctuation is retained.

[0094] Furthermore, all the processed text contents can be treated as a whole text information; or all the processed text contents can be segmented or word-segmented, thereby dividing a whole text information with more content into multiple text information segments with relatively less content.

[0095] It should be noted that the text information of this application can be used as a training sample for the aforementioned pre-trained large language model, and can also be used as a training sample for a text correction model.

[0096] Furthermore, the text sequence obtained above can be aligned with the text information to obtain the alignment result of the text sequence. It should be noted that the text alignment process between the text sequence and the text information can refer to the text alignment process between the recognized text segment and the corrected text.

[0097] Furthermore, based on the longest common subsequence between the text sequence and the text information in the alignment result, the timestamp information of the first common string in the text sequence compared with the text information in the video data can be determined, and the timestamp information of all strings from the first common string in the text sequence to the last string in the text sequence can be integrated, and the integrated timestamp information can be used as the group-level timestamp information of the text information.

[0098] Furthermore, a subtitle file can be generated based on the text information and its timestamp information. The specific process can refer to the above-mentioned process of generating a subtitle file based on the corrected text of each recognized text segment and its timestamp information.

[0099] The present application can perform text alignment on a text sequence with input text information and determine the timestamp information of the text information based on the alignment result, so that in a subtitle file generated based on the text information and its timestamp information, the subtitle text and audio are synchronized when displayed, thereby improving the display accuracy of the generated subtitles.

[0100] Compared with traditional subtitle generation methods, this application only requires or does not require manual provision of text information corresponding to the audio, which greatly reduces the time required to generate subtitles.

[0101] The subtitle generation device provided by the present application is described below. The subtitle generation device described below and the subtitle generation method described above can be referenced to each other.

[0102] Furthermore, the present application also provides a subtitle generating device.

[0103] The subtitle generating device comprises: A speech recognition module is used to perform speech recognition processing on the video data in the data to be processed, and obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; a text correction module, configured to perform text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; A text alignment module is used to align each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; an information determination module, configured to determine, based on an alignment result corresponding to each of the identified text segments and timestamp information of each character string in each of the identified text segments in the video data, timestamp information of a corrected text corresponding to each of the identified text segments; The subtitle generation module is used to generate a subtitle file based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0104] The subtitle generation device of the present application first performs speech recognition processing on the video data in the processing data to obtain the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; then, text correction is performed on each recognized text segment in the text sequence to obtain the corrected text of the corresponding recognized text segment, thereby effectively reducing the impact of the "hallucination" problem and improving the accuracy of the recognition result. Further, each recognized text segment in the text sequence is aligned with the corresponding corrected text to obtain the alignment result corresponding to each recognized text segment; and based on the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data, the timestamp information of the corrected text corresponding to the corresponding recognized text segment is determined respectively; because the text alignment is performed and the timestamp information of the corrected text is determined in combination with the alignment result, the subtitle text and the audio are synchronized when displayed in the subtitle file generated based on the corrected text corresponding to each recognized text segment and its timestamp information. Thus, the processing speed of subtitle generation is improved while the display accuracy is improved.

[0105] In one embodiment, the speech recognition module is specifically configured to: Performing filtering on the video data in the data to be processed to obtain filtered data; Performing voice activity detection on the filtered data to obtain a detection result; Segmenting the video data according to the detection result to obtain video segment data of corresponding data; Performing automatic speech recognition on each video clip data to obtain a recognition text segment corresponding to each video clip data and timestamp information of each character string in each recognition text segment in the video data; A text sequence of the video data is generated according to each recognized text segment.

[0106] In one embodiment, the text correction module is configured to perform the following operations for each recognized text segment in the text sequence when performing text correction for each recognized text segment in the text sequence: Encoding the current recognized text segment in the text sequence to obtain encoded data; Inputting the encoded data into a text correction model to obtain corrected encoded data output by the text correction model; wherein the text correction model is obtained by training a pre-trained large language model based on sample encoded data; the sample encoded data includes an encoding of a sample text and an encoding of a corrected version of the sample text; the sample text is any text containing an erroneous character string; The revised coded data is decoded to obtain a revised text of the current recognized text segment.

[0107] In one embodiment, the text alignment module is configured to perform the following operations for each recognized text segment in the text sequence when aligning each recognized text segment with the corresponding corrected text: The longest common subsequence of the current recognized text segment in the text sequence and the corrected text corresponding to the current recognized text segment are matched to obtain an alignment result corresponding to the current recognized text segment; the alignment result includes the longest common subsequence information of the recognized text segment and the corrected text, and the difference subsequence information between the recognized text segment and the corrected text.

[0108] In one embodiment, the subtitle generation module is specifically configured to: Determine subtitle file format information; The corrected text of each identified text segment and its timestamp information are used to generate a subtitle file according to the subtitle file format information.

[0109] In one embodiment, after obtaining the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data, if the data to be processed includes text information generated based on the content of the video data, the speech recognition module is further configured to: Performing text alignment on the text sequence and the text information to obtain an alignment result of the text sequence; determining the timestamp information of the text information according to the alignment result and the timestamp information of each character string in each recognized text segment of the text sequence in the video data; A subtitle file is generated based on the text information and its time stamp information.

[0110] Figure 2 An example of a physical structure diagram of an electronic device is shown below. Figure 2 As shown, the electronic device may include: a processor 210, a communication interface 220, a memory 230, and a communication bus 240, wherein the processor 210, the communication interface 220, and the memory 230 communicate with each other via the communication bus 240. The processor 210 may call logic instructions in the memory 230 to execute the following method: performing speech recognition processing on video data in the data to be processed, obtaining a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; Performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; Performing text alignment on each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; Determining the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data; A subtitle file is generated based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0111] In addition, the logic instructions in the aforementioned memory 230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0112] In another aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: performing speech recognition processing on video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; Performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; Performing text alignment on each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; Determining the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data; A subtitle file is generated based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0113] In another aspect, an embodiment of the present application further provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the methods provided in the above embodiments, for example, including: performing speech recognition processing on video data in the data to be processed, obtaining a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; Performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; Performing text alignment on each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; Determining the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data; A subtitle file is generated based on the corrected text corresponding to each of the identified text segments and its timestamp information.

[0114] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0115] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0116] Finally, it should be noted that the above embodiments are intended only to illustrate the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications, or equivalent substitutions of the technical solutions of the present application do not depart from the spirit and scope of the technical solutions of the present application.

Claims

1. A subtitle generation method, characterized in that: include: Performing speech recognition processing on the video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; Performing text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; Performing text alignment on each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; Determining the timestamp information of the corrected text corresponding to each recognized text segment according to the alignment result corresponding to each recognized text segment and the timestamp information of each character string in each recognized text segment in the video data; A subtitle file is generated based on the corrected text corresponding to each of the identified text segments and its timestamp information.

2. The subtitle generation method according to claim 1, wherein: When aligning each recognized text segment in the text sequence with the corresponding corrected text, the following operations are performed for each recognized text segment in the text sequence: The longest common subsequence of the current recognized text segment in the text sequence and the corrected text corresponding to the current recognized text segment are matched to obtain an alignment result corresponding to the current recognized text segment; the alignment result includes the longest common subsequence information of the recognized text segment and the corrected text, and the difference subsequence information between the recognized text segment and the corrected text.

3. The subtitle generation method according to claim 1, wherein: When performing text correction for each recognized text segment in the text sequence, the following operations are performed for each recognized text segment in the text sequence: Encoding the current recognized text segment in the text sequence to obtain encoded data; Inputting the encoded data into a text correction model to obtain corrected encoded data output by the text correction model; wherein the text correction model is obtained by training a pre-trained large language model based on sample encoded data; the sample encoded data includes an encoding of a sample text and an encoding of a corrected version of the sample text; the sample text is any text containing an erroneous character string; The revised coded data is decoded to obtain a revised text of the current recognized text segment.

4. The subtitle generation method according to claim 1, wherein: The performing speech recognition processing on the video data in the data to be processed to obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data includes: Performing filtering on the video data in the data to be processed to obtain filtered data; Performing voice activity detection on the filtered data to obtain a detection result; Segmenting the video data according to the detection result to obtain video segment data of corresponding data; Performing automatic speech recognition on each video clip data to obtain a recognition text segment corresponding to each video clip data and timestamp information of each character string in each recognition text segment in the video data; A text sequence of the video data is generated according to each recognized text segment.

5. The subtitle generation method according to claim 1, wherein: The generating of a subtitle file based on the corrected text corresponding to each identified text segment and its timestamp information includes: Determine subtitle file format information; The corrected text of each identified text segment and its timestamp information are used to generate a subtitle file according to the subtitle file format information.

6. The subtitle generation method according to claim 1, wherein: After obtaining the text sequence of the video data and the timestamp information of each character string in each recognized text segment contained in the text sequence in the video data, if the data to be processed includes text information generated based on the audio content of the video data, the method further includes: Performing text alignment on the text sequence and the text information to obtain an alignment result of the text sequence; determining the timestamp information of the text information according to the alignment result and the timestamp information of each character string in each recognized text segment of the text sequence in the video data; A subtitle file is generated based on the text information and its time stamp information.

7. A subtitle generating device, characterized in that: include: A speech recognition module is used to perform speech recognition processing on the video data in the data to be processed, and obtain a text sequence of the video data and timestamp information of each character string in each recognized text segment contained in the text sequence in the video data; a text correction module, configured to perform text correction on each recognized text segment in the text sequence to obtain a corrected text of the corresponding recognized text segment; A text alignment module is used to align each recognized text segment in the text sequence with the corresponding corrected text to obtain an alignment result corresponding to each recognized text segment; an information determination module, configured to determine, based on an alignment result corresponding to each of the identified text segments and timestamp information of each character string in each of the identified text segments in the video data, timestamp information of a corrected text corresponding to each of the identified text segments; The subtitle generation module is used to generate a subtitle file based on the corrected text corresponding to each of the identified text segments and its timestamp information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the subtitle generation method according to any one of claims 1 to 6 is implemented.

9. A storage medium, wherein the storage medium is a non-transitory computer-readable storage medium and stores a computer program, wherein: When the computer program is executed by a processor, the subtitle generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the subtitle generation method according to any one of claims 1 to 6 is implemented.

Images