Method for automatically generating subtitles and related products

Abstract

Embodiments of the present application provide a method for automatically generating subtitles and related products. The method comprises: obtaining audio information related to the timing of speech in a multimedia file; obtaining speech recognition text of the audio information; performing text readability enhancement processing on the speech recognition text; and generating subtitle text corresponding to the speech in the multimedia file based on the processed speech recognition text. Through the scheme of the present application, the subtitle text corresponding to the speech in the multimedia file can be automatically generated, which not only effectively saves the labor cost and time cost, but also ensures the high matching of the subtitle text and the speech. In addition, an apparatus and a computer readable storage medium are also provided.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of information processing technology, and more particularly, to a method for automatically generating subtitles, a device for executing the aforementioned method, and a computer readable storage medium. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application as set forth in the claims. Information in this section can include concepts that can not be previously conceived or explored. Therefore, unless otherwise indicated herein, information described in this section is not prior art to the present application and is not admitted to be prior art by its inclusion in this section.

[0003] In the process of listening learning, the listening material of a user can come from various channels. Therefore, there can be a situation that the listening material has no corresponding text or the listening material does not match the corresponding text (for example, the corresponding text is too long or the paragraph organization is unreasonable), which is not conducive to the user's learning by comparison. For this reason, artificial transcription of text information from audio and generation of corresponding text meeting the requirements as needed are usually used, which requires a large amount of time cost and labor cost. SUMMARY

[0004] The known generation process of the corresponding text of the listening material is time-consuming and labor-intensive, which is a very annoying process.

[0005] Therefore, there is a great need for an improved scheme for automatically generating subtitles and related products thereof, which can automatically generate subtitle text corresponding to the speech in a multimedia file, and the entire process does not require too much human intervention, effectively saving labor cost and time cost.

[0006] In this context, embodiments of the present application aim to provide a scheme for automatically generating subtitles and related products thereof.

[0007] In a first aspect of embodiments of the present application, a method for automatically generating subtitles is provided, comprising: obtaining audio information related to the time sequence of speech in a multimedia file; obtaining speech recognition text of the audio information; performing text readability enhancement processing on the speech recognition text; and generating subtitle text corresponding to the speech in the multimedia file based on the processed speech recognition text.

[0008] In an embodiment of the present application, obtaining audio information related to the time sequence of speech in a multimedia file comprises: performing sentence breaking processing on the speech in the multimedia file by using a front sentence breaking model to obtain the audio information.

[0009] In another embodiment of the present application, the punctuating the speech in the multimedia file using the pre-punctuation model comprises: predicting each audio frame in time sequence using the pre-punctuation model to obtain a punctuation label for each audio frame, wherein the punctuation label comprises four types: beginning-of-sentence silence, middle-of-sentence silence, end-of-sentence silence, and audio; and determining a first frame in the audio frame and of the type end-of-sentence silence as a punctuation point.

[0010] In yet another embodiment of the present application, the pre-punctuation model comprises a gated recurrent neural network model and a conditional random field decoding module, and the pre-punctuation model is pre-trained via: generating a training corpus from training audio with text annotation, wherein each audio frame in the training corpus is time-sequentially aligned with its corresponding text and each audio frame is annotated with a punctuation label; predicting each audio frame in time sequence using the gated recurrent neural network model and the conditional random field decoding module to obtain a predicted punctuation label; and training the pre-punctuation model based on the annotated punctuation label and the predicted punctuation label.

[0011] In still another embodiment of the present application, further comprising: in response to the audio information exceeding a predetermined time length, re-punctuating the audio information.

[0012] In one embodiment of the present application, the text readability enhancement processing on the speech recognition text comprises one or more of: format adjustment on the speech recognition text comprising at least adding punctuation; and punctuating the speech recognition text and the audio information using a post-punctuation model.

[0013] In another embodiment of the present application, the post-punctuation model comprises a time convolutional neural network model with an attention mechanism and a conditional random field decoding module, and the punctuating the speech recognition text and the audio information using the post-punctuation model comprises: encoding the audio information using the time convolutional neural network model; fusing the encoded output of the time convolutional neural network model and the speech recognition text corresponding to the audio information using the attention mechanism to obtain fused information; decoding the fused information using the conditional random field decoding module to obtain a punctuation label for each audio frame in the audio information, wherein the punctuation label comprises four types: beginning-of-sentence silence, middle-of-sentence silence, end-of-sentence silence, and audio; and determining a first frame in the audio frame and of the type end-of-sentence silence as a punctuation point.

[0014] In yet another embodiment of the present application, the format adjustment on the speech recognition text comprises at least adding punctuation, which comprises: performing written form adjustment on the speech recognition text; and performing punctuation and / or capitalization format adjustment on the written form adjusted speech recognition text using a format adjustment model.

[0015] In still another embodiment of the present application, wherein the format adjustment model comprises an encoder, a punctuation decoder and a capitalization decoder, the performing punctuation and capitalization format adjustment on the written form adjusted speech recognition text using the format adjustment model comprises: encoding the speech recognition text using the encoder; decoding the encoding output of the encoder using the punctuation decoder to obtain a punctuation output; and decoding the encoding output and the punctuation output using the capitalization decoder to obtain a capitalization output.

[0016] In yet another embodiment of the present application, the format adjustment model is pre-trained, and the training samples of the format adjustment model are obtained by: performing text alignment between a calibration text and a recognition text obtained by speech recognition; and performing punctuation calibration on the recognition text using the calibration text as the training sample.

[0017] In one embodiment of the present application, the performing written form adjustment on the speech recognition text comprises: constructing a text conversion model for text written conversion; converting spoken text in the speech recognition text using the text conversion model; and merging time stamps of the converted text to achieve text timing alignment.

[0018] In another embodiment of the present application, wherein the text conversion model comprises a weighted finite state machine, the converting spoken text in the speech recognition text using the text conversion model comprises: converting the spoken text in the speech recognition text using the weighted finite state machine, wherein the spoken text comprises at least one or more of the following: number text, money text, time text, date text, unit text and number text.

[0019] In yet another embodiment of the present application, the processed speech recognition text comprises one or more sentences, and the generating subtitle text based on the processed speech recognition text and the speech in the multimedia file comprises: determining the subtitle text based on the number of punctuation of a specified type and the number of words contained in each sentence.

[0020] In yet another embodiment of the present application, determining the subtitle text based on the number of punctuations of a specified type in each sentence and the number of words contained in the sentence comprises: in response to the number of punctuations being greater than a first threshold and the number of words being less than a second threshold, determining the subtitle text based on the sentence; or in response to the number of punctuations being greater than the first threshold and the number of words being greater than a third threshold, re-sentencing the sentence, and determining the subtitle text based on the result of re-sentencing.

[0021] In a second aspect of the embodiments of the present application, a device is provided, comprising: a processor; and a memory storing computer instructions for automatically generating a subtitle, which, when executed by the processor, cause the device to perform the method according to the foregoing and the following embodiments.

[0022] In a third aspect of the embodiments of the present application, a computer readable storage medium is provided, containing program instructions for automatically generating a subtitle, which, when executed by a processor, cause the device to perform the method according to the foregoing and the following embodiments.

[0023] According to the scheme for automatically generating a subtitle and the related products of the embodiments of the present application, the subtitle text corresponding to the speech in a multimedia file can be automatically generated without excessive human intervention. Specifically, the scheme of the present application obtains the speech recognition text of the audio information related to the timing of the speech, and obtains the corresponding subtitle text by using the text readability enhancement processing of the speech recognition text, which not only effectively saves the labor cost and time cost, but also ensures the high matching of the subtitle text and the speech. In some embodiments of the present application, the pre-sentence model can be used to predict and sentence process the speech in the multimedia file with multiple types of sentence tags, and combine multiple different types of sentence tags to accurately distinguish the silence at the beginning and end of the sentence, thereby effectively reducing the misjudgment probability in the sentence processing process. In other embodiments of the present application, the pre-trained format adjustment model can be used to adjust the punctuation of the speech recognition text, and the training samples of the format adjustment model are obtained by text alignment processing and punctuation calibration processing of the recognition text obtained by speech recognition and the calibration text, that is, the noise of speech misrecognition is introduced in the training process of the format adjustment model, which can effectively improve the robustness of the format adjustment model in the case of speech misrecognition.

[0024] In addition, in some other embodiments of the present application, a post-punctuation model is introduced to combine speech recognition text and audio information for punctuation processing, to realize multi-modal fusion of text and speech based on a time convolutional neural network model and attention mechanism, so as to further enhance the accuracy of punctuation. In addition, in some other embodiments of the present application, a conditional random field decoding module is introduced in the punctuation label type prediction process of the pre-punctuation model and the post-punctuation model, which can greatly enhance the smoothness and robustness of the prediction. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which a number of embodiments of the present application are shown by way of example, in which:

[0026] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present application is shown schematically;

[0027] Figure 2 A flowchart of a method for automatically generating subtitles according to one embodiment of the present application is shown schematically;

[0028] Figure 3 A flowchart of a method for automatically generating subtitles according to another embodiment of the present application is shown schematically;

[0029] Figure 4 A training framework diagram of a pre-punctuation model according to an embodiment of the present application is shown schematically;

[0030] Figure 5 A processing flowchart of speech punctuation in a multimedia file based on a pre-punctuation model according to an embodiment of the present application is shown schematically;

[0031] Figure 6 A text conversion process diagram based on a weighted finite state machine according to an embodiment of the present application is shown schematically;

[0032] Figure 7 A training framework diagram of a punctuation model according to an embodiment of the present application is shown schematically;

[0033] Figure 8 A training framework diagram of a post-punctuation model according to an embodiment of the present application is shown schematically;

[0034] Figure 9 A processing flowchart of speech punctuation in a multimedia file based on a post-punctuation model according to an embodiment of the present application is shown schematically; and

[0035] Figure 10A schematic block diagram of a device according to an embodiment of the present invention is shown.

[0036] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0037] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0038] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1 The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0039] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," "unit," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0040] Any combination of one or more computer readable medium can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0041] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0042] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0043] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In an embodiment of the application, the remote computer can be a server or another desktop computer.

[0044] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0045] These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to operate in a

[0046] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0047] According to embodiments of the present application, a method for automatically generating subtitles and related products are provided. In addition, any number of elements in the drawings are intended to represent example of a quantity of items, and not a limitation, and any designation of a term is intended to distinguish, but not limit.

[0048] The principles and spirit of the present application will be explained in detail below with reference to several representative embodiments of the present application. SUMMARY

[0050] The present inventors have found that the existing generation of a hearing material's reference text is time-consuming and labor-intensive. For example, the prior art is limited to relying on manual entry of text information and sentence breaking processing. This way of manually generating a reference text requires a large amount of time and labor costs, and cannot be applied to actual needs.

[0051] Based on this, the present inventors have found that the speech recognition text of audio information related to the timing of speech can be used to generate a subtitle text. Thus, without excessive human intervention, a subtitle text related to the timing of speech and reasonable sentence breaking can be generated.

[0052] In addition, the inventors have found that a voice activity detection model (VAD) commonly used in speech recognition technology can only distinguish between silence and non-silence, and cannot distinguish between mid-sentence silence and end-sentence silence, which can cause a certain degree of misjudgment. For example, mid-sentence silence is cut off, or end-sentence silence is too short and is not cut off. Based on this, the inventors have found that fine division of audio punctuation tags can be used to achieve accurate punctuation, thereby effectively reducing the probability of misjudgment in the punctuation process.

[0053] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail below.

[0054] Exemplary methods

[0055] The method for automatically generating subtitles according to exemplary embodiments of the present application will be described below with reference to Figure 2 It should be noted that embodiments of the present application can be applied to any applicable scenario.

[0056] Figure 2 A flowchart of a method 200 for automatically generating subtitles according to an embodiment of the present application is schematically shown. As Figure 2 indicated, at step S201, audio information related to the timing of speech in a multimedia file can be obtained. It should be noted that the multimedia file herein can include a song, an audio accompanying a book, a story audio, a film and television audio, or other audio and video files with subtitle output requirements, etc. The aforementioned audio information can include audio to be recognized related to the timing and order (e.g., time alignment and order alignment) of speech in the aforementioned multimedia file.

[0057] Next, at step S202, speech recognition text of the aforementioned audio information can be obtained. In some embodiments, the aforementioned audio information can be converted into speech recognition text using speech recognition technology (ASR). The ASR technology can automatically convert the user's speech content into corresponding text using a machine. It should be noted that the ASR technology is only used as an example herein, and the present application is not limited thereto. For example, other technologies capable of automatically converting speech into text can also be used.

[0058] Next, at step S203, the aforementioned speech recognition text can be subjected to text readability enhancement processing. The readability enhancement processing herein can be understood as processing that can increase the readability of the speech recognition text, such as the addition of punctuation, font format adjustment (e.g., case, italic, bold, etc.), text formalization adjustment, or punctuation, etc. It should be noted that the description of the readability enhancement processing herein is only exemplary, and the present application is not limited thereto.

[0059] Next, in step S204, subtitle text corresponding to the audio in the multimedia file can be generated based on the processed speech recognition text. Thus, the entire subtitle text generation process requires minimal manual intervention. It can obtain the corresponding subtitle text by acquiring speech recognition text containing audio information related to the speech timing and by enhancing the readability of that speech recognition text. This not only effectively saves manpower and time costs but also ensures a high degree of matching between the subtitle text and the audio.

[0060] Figure 3 A flowchart of a method 300 for automatically generating subtitles according to another embodiment of the present invention is illustrated schematically. It should be noted that method 300 can be understood as... Figure 2 Further clarification and supplementation of each step in the process. Therefore, the preceding text, combined with... Figure 2 The same description applies to the following text.

[0061] like Figure 3 As shown, in step S301, pre-segmentation processing can be performed on the speech in the multimedia file. As mentioned earlier, the multimedia file may include various types of audio and video files that require subtitle output. Specifically, pre-segmentation model and / or forced segmentation can be used to perform pre-segmentation processing on the speech in the multimedia file. For example, in some embodiments, the pre-segmentation model can be used to perform preliminary segmentation processing on the speech in the multimedia file. Then, audio information that still exceeds a predetermined duration after preliminary segmentation processing is segmented again to ensure that the audio information does not exceed the predetermined duration. It should be noted that the predetermined duration can be adjusted according to actual design requirements (e.g., 20s). In other embodiments, if the audio information obtained after preliminary segmentation processing by the pre-segmentation model does not exceed the predetermined duration, then forced segmentation is not necessary. In still other embodiments, forced segmentation processing can be performed first, followed by further segmentation processing in conjunction with the pre-segmentation model.

[0062] In some embodiments, the pre-segmentation model in the context of the present invention can be pre-trained. Figure 4 Figure 400 schematically illustrates a feasible training architecture for the pre-segmentation model. For example... Figure 4 As shown, the pre-segmentation model may include a gated recurrent neural network (GRU) model and a conditional random field (CRF) decoding module. Training this pre-segmentation model may involve generating training corpus based on text-annotated training audio. Then, using the GRU and CRF, frame-by-frame temporal prediction can be performed on each audio frame to obtain predicted segmentation labels. Finally, the pre-segmentation model is trained based on the annotated and predicted segmentation labels.

[0063] Specifically, the training audio Dtrain can be text-labeled, and then forced-aligned using speech recognition technology ASR to obtain aligned training corpus Xt of audio and corresponding text in time dimension. In addition, each audio frame in the training corpus Xt can be labeled with a punctuation label Yt. The punctuation label types can include beginning-of-sentence silence (0), middle-of-sentence silence (2), end-of-sentence silence (3), and audio (1). It should be noted that the division of the types of punctuation labels and the corresponding representative symbols are only exemplary, and the scheme of the present application is not limited thereto. Then, the gated recurrent neural network model GRU performs frame-by-frame prediction on each audio frame in the training corpus Xt in time sequence to obtain an intermediate variable ht. After the intermediate variable ht is processed by the conditional random field decoding module CRF, a predicted punctuation label Based on the labeled punctuation label Yt and the predicted punctuation label to obtain a loss function Loss, and the pre-punctuation model is trained using the loss function Loss.

[0064] Further, in some embodiments, the pre-punctuation model can be trained using Figure 4 The pre-punctuation model trained can be used to perform punctuation processing on the speech in the multimedia file. Figure 5 A feasible punctuation processing method 500 using the pre-punctuation model is schematically shown. As shown in Figure 5 At step S501, the pre-punctuation model can be used to perform frame-by-frame prediction on the speech in the multimedia file in time sequence to obtain a punctuation label for each audio frame. Specifically, the punctuation label for each audio frame can be any of beginning-of-sentence silence, middle-of-sentence silence, end-of-sentence silence, and audio. Then, at step S502, the first frame in the audio frame that is of the end-of-sentence silence type can be determined as a punctuation point. For example, a punctuation label for a segment of "0011211133" of audio frames is obtained by the pre-punctuation model described above, and the 9th frame in the segment of audio frames is the first frame of end-of-sentence silence (3). At this time, the 9th frame in the segment of audio frames can be taken as a punctuation point.

[0065] After the pre-punctuation processing is completed, return Figure 3 to step S302, the audio information after pre-punctuation processing can be subjected to speech recognition to obtain speech recognition text. As described above, the speech recognition can employ ASR speech recognition technology or other technologies capable of converting between speech and text.

[0066] Next, at step S303, text readability enhancement processing can be performed. For example, at step S303-1, spoken text in the speech recognition text can be converted into written text. Next, at step S303-2, punctuation and / or capitalization formatting adjustment can be performed on the written-form adjusted speech recognition text. Further, at step S303-3, a post-punctuation model can be used to perform punctuation processing in combination with the aforementioned speech recognition text and audio information. It should be noted that the execution order of steps S303-1, S303-2, and S303-3 is not limited herein, and can be adjusted according to actual design requirements. For example, S303-3 can be processed in parallel with S303-1 and S303-2.

[0067] In some embodiments, a text conversion model can be used to convert spoken text into written text. Figure 6 An exemplary conversion process 600 using a text conversion model is shown schematically. The text conversion model can employ a weighted finite state machine. Specifically, a weighted finite state machine (Weighted Finite-State Transducers, hereinafter referred to as WFST) can be constructed according to text de-regularization rules of the spoken text. Next, the input text is converted using the WFST. As shown in Figure 6 Each dashed box represents a basic finite state machine unit. When a piece of text is input, the text is automatically converted according to a weight sum and a minimum path. Therefore, when the input is "one hundred andthirteen", the text is converted to "113" (weight sum is 1.1), rather than "100and 13" (weight sum is 12.2). Then, the timestamps of the converted text can be merged to achieve text timing alignment. For example, the timestamps of the four words "one hundred and thirteen" can be merged into one to align with the text "113". The spoken text herein includes, but is not limited to, one or more of numerical text, money text, time text, date text, unit text, and numbered text.

[0068] It should be noted that the text conversion model can also be implemented by an end-to-end model of a Seq2Seq architecture, which involves an encoder / decoder that can be a recurrent neural network model or a Transformer type model. When the training data is sufficient, using an end-to-end model for text de-regularization generally has better results than WFST, especially in the case of misrecognition noise input.

[0069] In some embodiments, a format adjustment model can be used to perform punctuation and / or capitalization formatting adjustment on the written-form adjusted speech recognition text.Figure 7 A feasible training framework 700 for this format adjustment model is schematically illustrated. (e.g.) Figure 7 As shown, the format adjustment model can include an encoder, a punctuation decoder, and a case decoder. The encoder can include attention-based models such as recurrent neural networks, Transformer, and BERT. The punctuation and case decoders can include models such as multilayer perceptrons and conditional random fields. When training the format adjustment model, training samples X from the training dataset Dtrain are fed into the encoder to obtain the encoded vector H. Then, the encoded vector H is input into the punctuation decoder. Next, the output of the punctuation decoder and the encoded vector H are concatenated and input into the case decoder. The punctuation decoder and the case decoder output the punctuation and case prediction sequences ypunct and ycapt for each word in the sentence, respectively. Then, the punctuation and case prediction sequences ypunct and ycapt are used with the training dataset Ypunct and Ycapt to calculate the loss values ​​Lpunct and Lcapt. Backpropagation is then used to adjust the parameters, iterating repeatedly until the loss values ​​converge to train the format adjustment model.

[0070] In some embodiments, text alignment can be performed between the calibration text and the recognized text obtained through speech recognition. The recognized text can be obtained using ASR technology, and text alignment methods such as edit distance algorithms can be used to align the calibration text and the ASR-recognized text. Next, the calibration text can be used to calibrate the punctuation of the recognized text, serving as the aforementioned training sample X. Thus, by restoring correct punctuation to the recognized text, the consistency between the input data for model training and actual usage scenarios can be maximized. Using this training sample to train the format adjustment model significantly enhances its robustness in cases of ASR technology misrecognition.

[0071] In some embodiments, the encoder (e.g., a recurrent neural network model, Transformer, BERT, etc.) in the aforementioned trained format adjustment model can be used to encode the speech recognition text. Then, a punctuation decoder (e.g., a multilayer perceptron, conditional random field, etc.) can be used to decode the encoder's output to obtain punctuation output. A case decoder (e.g., a multilayer perceptron, conditional random field, etc.) can also be used to decode both the encoded output and the punctuation output to obtain uppercase and lowercase output. Thus, punctuation and case adjustment of the speech recognition text are achieved.

[0072] Furthermore, in some embodiments, a post-segmentation model can be used to perform segmentation processing by combining the aforementioned speech recognition text and audio information. Figure 8An example training framework 800 of the post-punctuation model is shown schematically. As shown in Figure 8 The post-punctuation model can include a temporal convolutional neural network model TCN with an attention mechanism and a conditional random field decoding module CRF, as shown. The attention mechanism can be an attention mechanism (ATT for short).

[0073] The temporal convolutional neural network model TCN then performs frame-by-frame prediction on each audio frame in the training corpus X to obtain an intermediate variable htcn. The ATT can then be used to fuse the intermediate variable htcn and the annotated text Text to output a hidden variable hatt. After processing by the conditional random field decoding module CRF, the predicted punctuation label Based on the annotated punctuation label Y and the predicted punctuation label The loss function Loss is obtained, and the post-punctuation model is trained using the loss function Loss.

[0074] Further, in some embodiments, the post-punctuation model can be trained using Figure 8 The post-punctuation model trained can be used for punctuation processing. Figure 9 An example punctuation processing method 900 using the post-punctuation model is shown schematically. As shown in Figure 9 At step S901, the audio information can be encoded using the aforementioned temporal convolutional neural network model (e.g., TCN). At step S902, the encoded output of the temporal convolutional neural network model and the speech recognition text corresponding to the audio information can be fused using an attention mechanism (e.g., ATT) to obtain fused information. At step S903, the aforementioned fused information can be decoded using a conditional random field decoding module (e.g., CRF) to obtain punctuation labels for each audio frame in the audio information. As mentioned earlier, the punctuation labels can include four types: sentence-initial silence, intra-sentence silence, sentence-final silence, and audio. At step S904, the first frame of each predetermined length of audio frames that is of the type sentence-final silence can be determined as a punctuation point. Thus, by introducing the post-punctuation model to combine speech recognition text and audio information for punctuation processing, the multi-modal fusion of text and speech based on the combination of the temporal convolutional neural network model and the attention mechanism is achieved, thereby further enhancing the accuracy of punctuation.

[0075] After the readability enhancement processing of the speech recognition text is completed, return to step S302. Figure 3 Then, the subtitle text can be determined according to the number of punctuation marks of a specified type and the number of words within the punctuation marks in each sentence break at step S304. The punctuation marks of the specified type can include a period and punctuation marks of the same level as the period (for example, exclamation marks, question marks, and the like that can distinguish different complete semantics). Specifically, if the number of punctuation marks of the specified type is detected to be greater than a first threshold, the number of words under each such punctuation mark can be further detected. If the number of words is less than a second threshold, it means that the number of words is small, and in this case, the subtitle text can be determined based on the sentence break. For example, the sentence break of "OK! I'll be there soon." contains two complete sentences, but the number of words in each sentence is not large, and in this case, no further sentence break processing is required. The sentence break result can be maintained and used as the subtitle text. If the number of words is greater than a third threshold, it means that the number of words is large, and in this case, the sentence break can be further processed to determine the subtitle text based on the result of the further sentence break. Thus, the readability of the subtitle text is further enhanced, and the reading experience of the user is improved. The first threshold, the second threshold, and the third threshold can be adjusted according to actual design requirements.

[0076] Based on this, the user can be helped to solve input problems in various use scenarios (such as hearing education and learning, etc.). In particular, in some scenarios, the user only has audio input, and through the scheme of the present application, the subtitle text can be efficiently obtained, and the time cost and labor cost of manually producing the subtitle text can be reduced. At the same time, compared with the traditional manual production method, the automatic generation of subtitles in the present application can more efficiently solve the needs of hearing in single-line text viewing or following, and the needs of repeating a single sentence. For example, it is not limited to the type of hearing or audio file (such as songs, audio accompanying books, story audio, audio of TV series, etc.), and can process and output corresponding subtitle files for these multimedia files. In addition, the generated subtitle file can be reasonably and accurately broken up, which facilitates the user to practice associated with the hearing. For example, the user needs to practice intensively (such as following reading, repeatedly listening to the current sentence, etc.) against a certain text. The conversion of spoken language and written language in the speech recognition process can also be intelligently processed, such as the number one hundred and thirteen displayed as 113, so as to be more easily read and compared.

[0077] Exemplary apparatus

[0078] After the method of the exemplary embodiments of the present application is introduced, next, with reference to Figure 10 The product related to the automatic generation of subtitles in the exemplary embodiments of the present application is described.

[0079] Figure 10 A schematic block diagram of the apparatus 1000 according to an embodiment of the present application is shown. As shown, the apparatus 1000 can include a processor 1001 and a memory 1002. The memory 1002 stores computer instructions for automatically generating a subtitle, which, when executed by the processor 1001, causes the apparatus 1000 to perform the method according to the foregoing description. Figure 10 Figure 2 Figure 3 For example, in some embodiments, the apparatus 1000 can perform the acquisition of audio information, the acquisition of speech recognition text, the generation of subtitle text, etc. Based on this, the apparatus 1000 can automatically generate a subtitle text corresponding to the speech in a multimedia file without human intervention.

[0080] In some implementation scenarios, the apparatus 1000 can include a device (such as a mobile phone, a PC, various smart electronic products, etc.) having a multimedia file input and speech information processing function and a subtitle display function. In actual applications, the apparatus 1000 can be constituted by a device having the above-mentioned multiple functions, or can be constituted by a combination of multiple devices having partial functions. The scheme of the present application does not limit the structural design that the apparatus 1000 can have.

[0081] It should be noted that although several devices or sub-devices for automatically generating a subtitle are mentioned in the foregoing detailed description, such division is merely not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided into multiple devices for embodiment.

[0082] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps.

[0083] The use of the verb "comprise", "comprising", and words of similar meaning in the application file does not exclude the presence of elements or steps other than those stated in the application file. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0084] ​​While the principles and spirit of the application have been described with reference to several specific embodiments, it is to be understood that the application is not limited to the specific embodiments disclosed, and that the division of the aspects is not meant to imply that features from these aspects cannot be combined to benefit, but is merely for ease of presentation. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the claims appended hereto. The scope of the claims appended is to be construed in the broadest sense to encompass all such modifications and equivalent structures and functions.

Claims

1. A method of automatically generating subtitles, characterized by, The method comprises: obtaining audio information related to the timing of speech in a multimedia file, the audio information comprising audio to be recognized related to the timing and order of speech in the multimedia file, and performing sentence breaking processing on the speech in the multimedia file using a pre-sentence breaking model to obtain the audio information, wherein performing sentence breaking processing on the speech in the multimedia file using a pre-sentence breaking model comprises: performing frame-by-frame prediction on the timing of the speech in the multimedia file using the pre-sentence breaking model to obtain a sentence breaking label for each audio frame, wherein the sentence breaking label comprises four types: sentence beginning silence, sentence middle silence, sentence end silence, and audio; and determining the first frame in the audio frame and of the type of sentence end silence as a sentence breaking point; obtaining speech recognition text of the audio information; performing text readability enhancement processing on the speech recognition text; and generating subtitle text corresponding to the speech in the multimedia file based on the processed speech recognition text; wherein the pre-sentence breaking model comprises a gated recurrent neural network model and a conditional random field decoding module, and the pre-sentence breaking model is pre-trained via the following operations: generating training corpus from training audio with text annotation, wherein each audio frame in the training corpus is aligned with its corresponding text timing and each audio frame is annotated with the type of sentence breaking label; performing frame-by-frame prediction on the timing of each audio frame using the gated recurrent neural network model and the conditional random field decoding module to obtain a predicted sentence breaking label; and training the pre-sentence breaking model based on the annotated sentence breaking label and the predicted sentence breaking label.

2. The method of claim 1, wherein, Further comprising: in response to the audio information exceeding a predetermined duration, performing re-sentence breaking processing on the audio information.

3. The method of claim 1, wherein, The text readability enhancement processing on the speech recognition text comprises one or more of the following: format adjustment of the speech recognition text at least comprising adding punctuation; and performing sentence breaking processing on the speech recognition text and the audio information using a post-sentence breaking model.

4. The method of claim 3, wherein, The post-sentence breaking model comprises a time convolutional neural network model with attention mechanism and a conditional random field decoding module, and performing sentence breaking processing on the speech recognition text and the audio information using a post-sentence breaking model comprises: encoding the audio information using the time convolutional neural network model; fusing the encoding output of the time convolutional neural network model and the speech recognition text corresponding to the audio information using the attention mechanism to obtain fusion information; decoding the fusion information using the conditional random field decoding module to obtain a sentence breaking label for each audio frame in the audio information, wherein the sentence breaking label comprises four types: sentence beginning silence, sentence middle silence, sentence end silence, and audio; and determining the first frame in each segment of audio frames of a predetermined duration and of the type of sentence end silence as a sentence breaking point.

5. The method of claim 3, wherein, The format adjustment of the speech recognition text at least comprising adding punctuation comprises: written form adjustment of the speech recognition text; and The format adjustment model is used to adjust the punctuation and / or capitalization of speech recognition text that has been formatted into written form.

6. The method of claim 5, wherein, The format adjustment model includes an encoder, a punctuation decoder, and a case decoder. The process of adjusting punctuation and case in the written-formatted speech recognition text using the format adjustment model includes: The encoder is used to encode the speech recognition text; The punctuation decoder is used to decode the encoder's output to obtain punctuation output; and The uppercase and lowercase outputs are decoded using the case decoder to obtain the uppercase and lowercase outputs.

7. The method of claim 6, wherein, The format adjustment model is pre-trained, and its training samples are obtained as follows: Align the calibration text with the recognized text obtained through speech recognition; and The recognized text is punctuated using calibration text as the training sample.

8. The method of claim 5, wherein, Adjusting the speech-recognized text to a written form includes: Construct a text conversion model for text-to-text conversion; The text conversion model is used to convert colloquial text in the speech recognition text; and The timestamps of the converted text are merged to achieve text time sequence alignment.

9. The method of claim 8, wherein, The text conversion model mentioned above includes a weighted finite state machine. The process of converting colloquial text in the speech recognition text using the text conversion model includes: The weighted finite state machine is used to convert the colloquial text in the speech recognition text into written text, wherein the colloquial text includes at least one or more of the following: numerical text, monetary text, time text, date text, unit text, and numbered text.

10. The method according to any one of claims 1 to 9, characterized in that, The processed speech recognition text includes one or more sentence segments. Generating subtitle text based on the processed speech recognition text that corresponds to the speech in the multimedia file includes: The subtitle text is determined based on the number of punctuation marks of a specified type in each sentence and the number of words contained therein.

11. The method of claim 10, wherein, The subtitle text is determined based on the number of punctuation marks of a specified type and the number of words contained in each sentence segment, including: In response to the number of punctuation marks being greater than a first threshold and the number of words being less than a second threshold, the subtitle text is determined based on the sentence segmentation; or In response to the number of punctuation marks being greater than a first threshold and the number of words being greater than a third threshold, the sentence segmentation is performed again, and the subtitle text is determined based on the result of the second sentence segmentation.

12. An apparatus, characterized by include: processor; as well as A memory storing computer instructions for automatically generating subtitles, which, when executed by the processor, cause the device to perform the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The program includes instructions for automatically generating subtitles, which, when executed by a processor, cause the method according to any one of claims 1-11 to be implemented.

Images