A method, device and medium for extracting subtitles from teaching videos

Abstract

The present invention provides a method for extracting subtitles from teaching videos, comprising: obtaining voice data of the teaching video, and preprocessing the voice data according to the audio features of the corresponding time segment, classifying the teaching video into subtitled video segments and non-subtitled video segments; generating and correcting text labels for the voice data of the subtitled video segments, establishing a corresponding voice data set containing text labels, and optimizing a whisper model; using the optimized whisper model to transcribe the voice data to generate a labeling file, using Prompt to splice the corresponding proprietary vocabulary and / or hot words with the labeling file, and generating corresponding subtitles; matching the subtitles with the corresponding video according to the audio features of the time segment, and outputting the subtitles in the subtitle area. The present invention also exemplarily discloses a device and storage medium for extracting subtitles from teaching videos.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video intelligent analysis, and in particular to a method and device for extracting subtitles based on teaching videos and a computer-readable storage medium. Background Art

[0002] With the booming development of online education, digital course resources and other fields, teaching videos have become a key carrier for knowledge dissemination, and the accuracy of subtitle extraction has become increasingly critical. Subtitle extraction technology has undergone multiple stages of evolution. Early rule-based methods relied on preset voice models, grammar rule libraries and dictionaries to analyze audio features and match words to generate subtitles. However, due to the flexibility of language and the difficulty of exhausting rules, they had poor adaptability to complex teaching content (including professional vocabulary and colloquial expressions), and subtitle errors frequently occurred.

[0003] Subsequently, the statistical machine learning stage emerged, using hidden Markov models (HMM) and Gaussian mixture models (GMM) to build acoustic models, combined with N-gram language models, and using a large amount of speech data training to improve the accuracy of subtitles. However, this method is highly dependent on the amount and quality of data, has limited generalization capabilities across fields (teaching videos of different subjects), and is slow to learn new vocabulary and professional terms.

[0004] At present, deep learning dominates the field of subtitle extraction, and many open source and commercial technologies have emerged. The most similar to the present invention is Whisper, which is an open source masterpiece of OpenAI. Based on the Transformer architecture, the encoder converts audio into a feature sequence, and the decoder predicts text based on the features and preset start symbols. It has been pre-trained with massive multilingual voice data and has demonstrated strong strength. Like common foreign language teaching videos, whether it is British or American English accents, or French or German teaching content, it can accurately transcribe most sentences; in Chinese teaching scenarios, it has a considerable accuracy rate in recognizing standard expressions in Mandarin, and can handle daily popular science and basic subject explanation videos with ease.

[0005] However, existing technologies still have shortcomings in extracting subtitles from teaching videos. Faced with the diverse and complex demands of teaching scenarios, it is difficult to fully guarantee high-accuracy subtitle output; in particular, it is difficult to handle proprietary and emerging vocabulary. Summary of the invention

[0006] The present invention provides a method and device for extracting subtitles based on teaching videos and a computer-readable storage medium, which utilizes a large model to extract subtitles for proprietary vocabulary and / or hot words, thereby improving the matching degree between subtitles and audio content, thereby improving the comprehensibility and usability of teaching videos and better serving educational and teaching activities.

[0007] In a first aspect, a method for extracting subtitles from a teaching video is provided, comprising: acquiring voice data of the teaching video, and preprocessing the voice data according to the audio features of the corresponding time segment, classifying the teaching video into a subtitled video segment and a non-subtitled video segment; for the voice data of the subtitled video segment, generating and correcting text labels, establishing a corresponding voice data set containing text labels, and optimizing a whisper model; using the optimized whisper model to transcribe the voice data to generate an annotation file, using Prompt to splice corresponding proprietary vocabulary and / or hot words with the annotation file to generate corresponding subtitles; matching the subtitles with the corresponding video according to the audio features of the time segment, and outputting the subtitles in a subtitle area.

[0008] In some embodiments, preprocessing is performed based on audio features of corresponding time segments to classify the teaching video into subtitled video segments and non-subtitled video segments, including: encoding the speech data into a word vector sequence; encoding and extracting features of the word vector sequence through a feature encoder; detecting and clustering the speech data and the teaching video, and classifying the teaching video into subtitled video segments and non-subtitled video segments.

[0009] In some embodiments, for the speech data of the subtitled video clip, text labels are generated and corrected, a corresponding speech data set containing text labels is established, and the whisper model is optimized, including: using a language information extraction network pre-trained with sample video text to generate and correct text labels, and establish a corresponding speech data set containing text labels; cutting the speech data set into multiple samples through text labels, inputting any sample into the whisper model, and optimizing model parameters.

[0010] In some embodiments, the voice data is transcribed using an optimized whisper model to generate an annotation file, and the corresponding proprietary vocabulary and / or hot words are spliced ​​with the annotation file using Prompt, including: calculating with characters as units and adding weights, using edit distance / element composition length as the similarity standard to calculate the similarity between each character in the word and each character in the proprietary vocabulary and / or hot words according to their element composition, and in the process, different weighted proportions are given considering the influence of different elements on the similarity; or, calculating with characters in the word as units, and the similarity calculation of the word needs to consider the extreme difference value of the similarity of all characters in the word, and after obtaining the similarity of each character in the word, the mean and the extreme difference value are calculated, and the mean plus the extreme difference value multiplied by a certain reduction coefficient is used as the similarity of the character; generating a token based on the similarity, and splicing the corresponding proprietary vocabulary and / or hot words with the annotation file using Prompt.

[0011] In some embodiments, the subtitles are matched with corresponding videos according to the audio features of the time segment, and the subtitles are output in a subtitle area, including: detecting and tracking the subtitles so that the subtitles appear and disappear within the time segment; based on the subtitle area coordinates, the subtitles are output and background areas and noise subtitles that are mistakenly identified as subtitles are filtered out.

[0012] In some embodiments, the speech data set is cut into multiple samples through text labels, any sample is input into the whisper model, and the model parameters are optimized, including: performing time-series average pooling on any sample output by the encoder of each layer of the whisper model; inputting any sample after time-series average pooling into the time-series transformer; performing time-series average pooling again after processing through the time-series transformer; splicing all samples after two time-series average pooling processes, and inputting them into the hierarchical transformer to optimize the model parameters.

[0013] In some embodiments, detecting and tracking subtitles includes: in the subtitle area, tracking along the time axis within the range of the time segment to obtain the appearance and disappearance time points of the subtitles.

[0014] In a second aspect, a device for extracting subtitles from teaching videos is provided, comprising: a data module, used to obtain voice data of the teaching video, and pre-process the voice data according to the audio features of the corresponding time segment, and classify the teaching video into subtitled video segments and non-subtitled video segments; a large model module, used to generate and correct text labels for the voice data of the subtitled video segments, establish a corresponding voice data set containing text labels, and optimize the whisper model; a processing module, used to transcribe the voice data using the optimized whisper model to generate an annotation file, use Prompt to splice the corresponding proprietary vocabulary and / or hot words with the annotation file, and generate corresponding subtitles; a display module, used to match the subtitles with the corresponding video according to the audio features of the time segment, and output the subtitles in the subtitle area.

[0015] In a third aspect, the present invention provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned teaching video subtitle extraction method is implemented.

[0016] In a fourth aspect, the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the above-mentioned teaching video subtitle extraction method.

[0017] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0018] First, for students, accurate subtitles can help them better understand the content of teaching videos, especially when it involves complex professional knowledge and a lot of new vocabulary. For example, when studying teaching videos of advanced mathematics courses, accurate subtitles can help students understand the derivation process of mathematical formulas and the meaning of related terms more clearly, improving learning effects.

[0019] Second, for teachers and educational institutions, high-quality subtitles can help improve the quality and professionalism of teaching videos and enhance the competitiveness of teaching resources.

[0020] Third, compared with the traditional method of manual proofreading and correction of subtitles, it can greatly reduce manpower input; there is no need to manually check and correct vocabulary recognition errors one by one, saving a lot of time and labor costs.

[0021] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0023] Figure 1 A flowchart of a method for extracting subtitles from a teaching video provided by an embodiment of the present application is shown;

[0024] Figure 2 is a schematic block diagram of a teaching video subtitle extraction device provided in an embodiment of the present application;

[0025] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of an application scenario of an embodiment of the present application. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0029] This application provides a method for extracting subtitles from teaching videos. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 A method for extracting subtitles from a teaching video is described in detail in the first embodiment of the present application.

[0030] The present application provides a method 100 for extracting subtitles from teaching videos. The processing flow of the method 100 may include the following steps:

[0031] Step S102: Acquire audio data of the teaching video, that is, acquire voice data of the teaching video, and pre-process the audio data according to the audio features of the corresponding time segment, and classify the teaching video into subtitled video segments and non-subtitled video segments.

[0032] In some embodiments, preprocessing is performed based on audio features of corresponding time segments to classify the teaching video into subtitled video segments and non-subtitled video segments, which may include: encoding the speech data into a word vector sequence; encoding and extracting features of the word vector sequence through a feature encoder; detecting and clustering the speech data and the teaching video, and classifying the teaching video into subtitled video segments and non-subtitled video segments.

[0033] Step S104: Optimizing the Whisper model, that is, generating and correcting text labels for the voice data of the subtitled video clips, establishing a corresponding voice data set containing text labels, and optimizing the whisper model. It should be noted that the Whisper of this embodiment supports speech recognition in multiple languages ​​and can process audio input in different languages. This makes it have wide application potential around the world. The code and model of Whisper are open source, and users can modify and optimize them according to their own needs. This enables developers to integrate the functions of Whisper in their own applications.

[0034] In some embodiments, for the speech data of the subtitled video clip, text labels are generated and corrected, a corresponding speech data set containing text labels is established, and the whisper model is optimized, which may include: using a language information extraction network pre-trained with sample video text to generate and correct text labels, and establish a corresponding speech data set containing text labels; cutting the speech data set into multiple samples through text labels, inputting any sample into the whisper model, and optimizing model parameters.

[0035] In some embodiments, the speech data set is cut into multiple samples through text labels, any sample is input into the whisper model, and the model parameters are optimized, including: performing time-series average pooling on any sample output by the encoder of each layer of the whisper model; inputting any sample after time-series average pooling into the time-series transformer; performing time-series average pooling again after processing through the time-series transformer; splicing all samples after two time-series average pooling processes, and inputting them into the hierarchical transformer to optimize the model parameters.

[0036] Step S106: Special words and / or hot words are spliced ​​with the annotation file to generate subtitles, that is, the voice data is transcribed using the optimized whisper model to generate an annotation file, and the corresponding special words and / or hot words are spliced ​​with the annotation file using Prompt to generate corresponding subtitles. Among them, hot words refer to words or phrases that frequently appear in a specific context, such as industry terms, brand names, character names, etc. By incorporating these words into the recognition system, the system's recognition rate of these words can be improved.

[0037] In some embodiments, the voice data is transcribed using an optimized whisper model to generate an annotation file, and the corresponding proprietary vocabulary and / or hot words are spliced ​​with the annotation file using Prompt, including: calculating with characters as units and adding weights, using edit distance / element composition length as the similarity standard to calculate the similarity between each character in the word and each character in the proprietary vocabulary and / or hot words according to their element composition, and in the process, different weighted proportions are given considering the influence of different elements on the similarity; or, calculating with characters in the word as units, and the similarity calculation of the word needs to consider the extreme difference value of the similarity of all characters in the word, and after obtaining the similarity of each character in the word, the mean and the extreme difference value are calculated, and the mean plus the extreme difference value multiplied by a certain reduction coefficient is used as the similarity of the character; generating a token based on the similarity, and splicing the corresponding proprietary vocabulary and / or hot words with the annotation file using Prompt.

[0038] In some embodiments, relevant proprietary words and hot words can be manually or automatically collected according to the subject of the teaching video and the domain knowledge that may be involved. For example, for mathematics teaching videos, key terms in mathematical formulas, names of mathematicians, etc. are collected; for language teaching videos, new popular words, grammatical terms, etc. are collected. These hot words are added to the input of the model in a specific format (such as the form of <code start> hotwords <code end> ​​parameters) so that they can be given priority by the model during the transcription process.

[0039] Step S108: outputting subtitles, that is, matching the subtitles with the corresponding video according to the audio features of the time segment, and outputting the subtitles in the subtitle area.

[0040] In some embodiments, the subtitles are matched with corresponding videos according to the audio features of the time segment, and the subtitles are output in a subtitle area, including: detecting and tracking the subtitles so that the subtitles appear and disappear within the time segment; based on the subtitle area coordinates, the subtitles are output and background areas and noise subtitles that are mistakenly identified as subtitles are filtered out.

[0041] In some embodiments, detecting and tracking subtitles includes: in the subtitle area, tracking along the time axis within the range of the time segment to obtain the appearance and disappearance time points of the subtitles.

[0042] In some embodiments, the generated subtitle text can be time-stamp aligned to ensure that the subtitles and the audio are accurately matched in time. At the same time, the subtitles are formatted to conform to common subtitle format standards, such as SRT format, to facilitate loading and display in a video player.

[0043] Figure 2 2 is a schematic block diagram of a teaching video subtitle extraction device 200 according to an exemplary embodiment. The teaching video subtitle extraction device 200 includes: a data module 202, a large model module 204, a processing module 206 and a display module 208, wherein:

[0044] The data module 202 is used to obtain the voice data of the teaching video, and pre-process it according to the audio features of the corresponding time segment, and classify the teaching video into subtitle video segments and non-subtitle video segments.

[0045] The large model module 204 is used to generate and correct text labels for the speech data of the subtitle video clip, establish a corresponding speech data set containing text labels, and optimize the whisper model.

[0046] The processing module 206 is used to transcribe the speech data into a marked file using the optimized whisper model, and use Prompt to concatenate the corresponding proprietary words and / or hot words with the marked file to generate corresponding subtitles.

[0047] The display module 208 is used to match the subtitles with the corresponding video according to the audio features of the time segment, and output the subtitles in the subtitle area.

[0048] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002 and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 3001, the steps of the above-mentioned teaching video subtitle extraction method are implemented.

[0049] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0050] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 3 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0051] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0052] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0053] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0054] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0055] It should be noted that Figure 3 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0056] like Figure 4 The application scenario diagram of the illustrated embodiment is a schematic diagram of an application scenario of the method for extracting subtitles from a teaching video provided by the embodiment of the present application. In the application scenario diagram, a terminal device 401 and a server 402 are included. The terminal device 401 and the server 402 communicate with each other via a communication network.

[0057] The terminal device 401 is an electronic device used by the target object, and the electronic device may be a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a vehicle-mounted terminal, etc. In addition, a client related to the teaching video subtitle extraction method may be installed on the terminal device 401, and the client may be software (for example, an APP, a browser, etc.), or a web page, a small program, etc. By implementing the teaching video subtitle extraction method on the terminal device 401, the related operations of the teaching video subtitle extraction method are performed.

[0058] Server 402 may be an independent physical server or an edge device in the field of cloud computing. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, cloud functions, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (English name: Content Delivery Network, abbreviated as CDN), as well as big data and artificial intelligence platforms.

[0059] There is no restriction on the number of the terminal devices 401 and / or servers 402 .

[0060] It should be noted that the method for extracting subtitles from teaching videos in the embodiment of the present application can be executed by the terminal device 401 or the server 402 alone, or can be executed jointly by the terminal device 401 and the server 402. For example, when executed jointly by the terminal device 401 and the server 402, the server 402 can provide the voice data of the teaching video to the terminal device 401, and pre-process the voice data according to the audio features of the corresponding time segment, and classify the teaching video into subtitled video segments and non-subtitled video segments. The terminal device 401 can provide the server 402 with a method for transcribing the voice data using the optimized whisper model to generate an annotation file, and use Prompt to splice the corresponding proprietary vocabulary and / or hot words with the annotation file to generate corresponding subtitles. According to the audio features of the time segment, the subtitles are matched with the corresponding video, and the subtitles are output in the subtitle area.

[0061] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned teaching video subtitle extraction method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

[0062] An embodiment of the present invention further provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned teaching video subtitle extraction method is implemented.

[0063] An embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the above-mentioned teaching video subtitle extraction method.

[0064] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0065] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0066] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0067] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0068] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0070] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0071] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0074] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0075] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for extracting subtitles from teaching videos, characterized in that: include: Acquire the voice data of the teaching video, and pre-process it according to the audio features of the corresponding time segment, and classify the teaching video into subtitle video segments and non-subtitle video segments; For the speech data of the subtitled video clip, text labels are generated and corrected, a corresponding speech data set containing text labels is established, and the whisper model is optimized, including: using the language information extraction network obtained by pre-training the sample video text, generating and correcting the text labels, and establishing a corresponding speech data set containing text labels; cutting the speech data set into a plurality of samples by the text labels, inputting any sample into the whisper model, and optimizing the model parameters, including: performing time series average pooling on any sample output by the encoders of each layer of the whisper model; inputting any sample after the time series average pooling into the time series transformer; performing time series average pooling again after being processed by the time series transformer; splicing all the samples processed by the above method, and inputting them into the hierarchical transformer to optimize the model parameters; The optimized whisper model is used to transcribe the speech data to generate an annotation file, and prompt is used to splice the corresponding proprietary vocabulary and / or hot words with the annotation file, including: taking characters as units for calculation and adding weights, and taking edit distance / element composition length as the similarity standard to calculate the similarity between each character in the word and each character in the proprietary vocabulary and / or hot words according to their element composition, and in the process, different weighting ratios are given considering the influence of different elements on the similarity; or, taking characters in the word as units for calculation, and the similarity calculation of the word needs to consider the extreme difference value of the similarity of all characters in the word, and after obtaining the similarity of each character in the word, the mean and the extreme difference value are calculated, and the mean plus the extreme difference value after multiplying by a certain reduction coefficient is used as the similarity of the character; generating a token according to the similarity, and using prompt to splice the corresponding proprietary vocabulary and / or hot words with the annotation file to generate corresponding subtitles; According to the audio features of the time segment, the subtitles are matched with the corresponding video, and the subtitles are output in a subtitle area.

2. The teaching video subtitle extraction method according to claim 1, characterized in that: Preprocessing is performed according to the audio features of the corresponding time segments to classify the teaching video into subtitled video segments and non-subtitled video segments, including: Encoding the speech data into a word vector sequence; Encode and extract features from word vector sequences through feature encoders; The voice data and the teaching video are detected and clustered, and the teaching video is classified into subtitled video segments and non-subtitled video segments.

3. The teaching video subtitle extraction method according to claim 1, characterized in that: According to the audio features of the time segment, the subtitles are matched with the corresponding video, and the subtitles are output in a subtitle area, including: Detecting and tracking subtitles so that the subtitles appear and disappear within the time segment; Based on the subtitle area coordinates, the subtitles are output and background areas and noise subtitles that are misjudged as subtitles are filtered out.

4. The teaching video subtitle extraction method according to claim 3, characterized in that: For subtitle detection and tracking, including: In the subtitle area, the subtitles are tracked along the time axis within the range of the time segment to obtain the appearance and disappearance time points of the subtitles.

5. A teaching video subtitle extraction device, characterized in that: include: A data module, used to obtain the voice data of the teaching video, and pre-process the voice data according to the audio features of the corresponding time segment, and classify the teaching video into subtitled video segments and non-subtitled video segments; The large model module is used to generate and correct text labels for the voice data of the subtitle video clip, establish a corresponding voice data set containing text labels, and optimize the whisper model, including: using the language information extraction network obtained by pre-training the sample video text to generate and correct text labels, and establish a corresponding voice data set containing text labels; cutting the voice data set into multiple samples by text labels, inputting any sample into the whisper model, and optimizing the model parameters, including: performing time-series average pooling on any sample output by the encoder of each layer of the whisper model; inputting any sample after the time-series average pooling into the time-series transformer; performing time-series average pooling again after being processed by the time-series transformer; splicing all the samples processed by the above methods, and inputting them into the hierarchical transformer to optimize the model parameters; A processing module is used to transcribe the voice data using the optimized whisper model to generate an annotation file, and use Prompt to splice the corresponding proprietary vocabulary and / or hot words with the annotation file, including: calculating with characters as units and adding weights, using edit distance / element composition length as a similarity standard to calculate the similarity between each character in the word and each character in the proprietary vocabulary and / or hot words according to their element composition, and in the process, different weighted proportions are given considering the influence of different elements on the similarity; or, calculating with characters in the word as units, and the similarity calculation of the word needs to consider the extreme difference value of the similarity of all characters in the word, and after obtaining the similarity of each character in the word, the mean and the extreme difference value are calculated, and the mean plus the extreme difference value multiplied by a certain reduction coefficient is used as the similarity of the character; generating a token according to the similarity, splicing the corresponding proprietary vocabulary and / or hot words with the annotation file using Prompt, and generating corresponding subtitles; The display module is used to match the subtitles with the corresponding video according to the audio features of the time segment and output the subtitles in a subtitle area.

6. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for extracting subtitles from a teaching video as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the teaching video subtitle extraction method as described in any one of claims 1 to 4.

Images