A method and system for automatically generating multilingual MOOCs based on recorded courses

Abstract

The present invention provides a method and system for automatically generating multilingual MOOCs based on recorded courses, and relates to the field of digital technology. The present invention realizes multi-dimensional intelligent arrangement of courses and multilingual video synthesis through the steps of intelligent noise reduction, speech recognition, speech recognition result correction, voiceprint training sample extraction, voiceprint model training and the like, and comprises an intelligent noise reduction module, an intelligent speech recognition module, an intelligent speech recognition result correction module, an intelligent voiceprint training sample extraction module, a voiceprint model training module and a multilingual video synthesis module. The present invention solves the problem that a large amount of manpower and material resources are consumed and rapid large-scale conversion is impossible when converting recorded courses into multilingual MOOC resources in the prior art. Recorded course videos can be automatically converted into MOOC videos in different languages ​​in batches without manual intervention, thereby improving the robustness of the voiceprint model, realizing the refined arrangement of course content, saving a large amount of manpower and material resources, and realizing rapid large-scale internationalization conversion of courses.

Description

Technical Field

[0001] The present invention relates to the field of digital technology, and in particular to a method and system for automatically generating multilingual MOOCs based on recorded courses. Background Art

[0002] Massive open online courses (MOOCs) are dedicated to helping everyone who is willing to improve themselves get better higher education for free. At present, China has become the world's largest in the construction and application of MOOCs. If we can gradually enrich the resource types and customize multilingual MOOC groups based on the full use of China's MOOC resources, we will be able to build two-way communication channels with more countries and online education institutions for high-quality courses, teaching materials, virtual simulation experiments and other online education resources, thereby realizing the vision of "MOOC going global" and promoting the exchange and sharing of high-quality resources internationally.

[0003] At present, colleges and universities are very rich in high-quality resources. Basically, all colleges and universities have established a regular course recording and broadcasting system. Every day, a large number of outstanding experts and professors are imparting knowledge to students. However, in order to convert teachers' courses into international MOOC resources, professional recording and post-production methods are still used. This requires a lot of manpower and material resources, and it is impossible to achieve rapid and large-scale internationalization of courses.

[0004] However, there are currently very few methods for automating the conversion of recorded courses. For example, the invention with publication number CN107633719A discloses a method for separating and synthesizing courses and speech. The method for separating and synthesizing courses and speech can effectively change the language type of the original video, reduce the impact of mouth movements in the original video on the subsequent rendering process, and improve the synchronization rate of audio and video; but there is no complete path and algorithm for solving how to effectively and automatically convert recorded courses into multilingual courses.

[0005] For example, the invention with publication number CN116312083A discloses a method for converting a single course language into multiple languages ​​and generating corresponding subtitles. This invention only solves the problem of converting speech into multiple language subtitles, but does not solve the problem of how to convert speech into multiple languages ​​and use the teacher's own voiceprint for pronunciation, and how to change the video when the length of speech changes into multiple languages.

[0006] Therefore, the present invention provides a method and system for automatically generating multilingual MOOCs based on recorded courses, so as to provide a complete multilingual MOOC generation solution. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and system for automatically generating multilingual MOOCs based on recorded courses, so as to provide a complete multilingual MOOC generation solution; the purpose of the present invention is achieved through the following technical solutions:

[0008] On the one hand, the present invention provides a method for automatically generating multilingual MOOCs based on recorded courses, and performs live teaching PPT alignment through the following steps:

[0009] Step A: Collect the regular recorded course video files of the current recorded course;

[0010] Step B: Perform intelligent noise reduction processing on the regular recorded course video files to obtain the noise-reduced course videos;

[0011] Step C: Perform intelligent speech recognition on the course video after noise reduction to obtain the corresponding speech recognition result;

[0012] Step D: performing intelligent speech recognition result correction on the speech recognition result to obtain a corresponding corrected speech recognition result;

[0013] Step E: Extract intelligent voiceprint training samples from the corrected speech recognition results to obtain corresponding voiceprint sample files;

[0014] Step F: Train the voiceprint model using the voiceprint sample file. After the training is completed, the teacher's voiceprint model is obtained and assigned a corresponding model ID;

[0015] Step G1: Perform multi-dimensional intelligent sorting of the course video after noise reduction and the corrected speech recognition result to obtain sorted course video and sorted speech recognition result;

[0016] Step G2: translating the sorted speech recognition results into multiple languages ​​to obtain corresponding translated texts;

[0017] Step G3: Perform TTS multi-language synthesis based on the translated text and the teacher's voiceprint model to obtain a synthesized new voice;

[0018] Step G4: The synthesized new voice and the sorted course video are synthesized to obtain a finished video translated into the corresponding language; wherein the synthesis process includes speed alignment and subtitle addition.

[0019] As a further solution, in step A, image area extraction is also performed on the regular recorded course video file; wherein the image area includes the PPT screen area, the blackboard area and the teacher image area.

[0020] As a further solution, in step D, the intelligent speech recognition result correction is performed through the following steps:

[0021] Perform OCR recognition on the PPT screen area of ​​the video file to obtain OCR recognized text;

[0022] Through NLP analysis of OCR recognized text, correction terms are extracted and a PPT keyword library corresponding to the current recorded course is created; correction terms include important nouns, proper nouns and high-frequency nouns;

[0023] Regular expressions detect typos or commonly misrecognized words in speech recognition results, and compare similarities with correction terms in the PPT keyword library;

[0024] If the similarity reaches the correction judgment threshold, the corresponding typos or commonly misrecognized words are automatically replaced with correction terms in the PPT keyword library.

[0025] As a further solution, in step E, intelligent voiceprint training sample extraction is performed through the following steps:

[0026] Sentence segmentation is performed on the corrected speech recognition result to obtain a number of sentence units and the saliva words are filtered out through the stop saliva word library;

[0027] Score the phoneme diversity of the sentence unit to obtain the phoneme diversity score of the corresponding sentence unit;

[0028] Score the emotional stability of the sentence unit to obtain the emotional stability score of the corresponding sentence unit;

[0029] Sentence length scoring is performed on the sentence unit to obtain the sentence length score of the corresponding sentence unit;

[0030] Scoring the background noise of the sentence unit to obtain the background noise score of the corresponding sentence unit;

[0031] Score the speech speed of the sentence unit to obtain the speech speed score of the corresponding sentence unit;

[0032] Based on the phoneme diversity score, emotional stability score, sentence length score, background noise score and speech speed score, each sentence unit is comprehensively scored to obtain a comprehensive score of the sentence unit; the comprehensive score adopts a comprehensive scoring mechanism with adaptive weights;

[0033] The sentence units are sorted according to the comprehensive scores, and sentence units with a set ratio are selected from high to low, spliced ​​into a complete audio file, and used as a voiceprint sample file.

[0034] As a further solution, the sentiment stability score is calculated through the following steps: the sentence unit is divided into several speech segments by word; the sentiment analysis model BERT is used to calculate the sentiment intensity value of each speech segment; the standard deviation V of the sentiment fluctuation is calculated; the sentiment stability score is calculated by the sentiment score formula ;in, , K It is the magnification factor used to adjust the score impact;

[0035] The phoneme diversity score is performed by the following steps: extracting the phoneme sequence of the sentence unit; identifying the number of unique phonemes and the total number of phonemes in the phoneme sequence; calculating the phoneme diversity score ;in, ;

[0036] Sentence length score: obtained by setting the ideal length range and calculating the deviation of the sentence length range; Background noise score: evaluated by the signal-to-noise ratio (SNR) to assess the impact of background noise on sentence quality; Speech speed score: obtained by setting the ideal speech speed range and calculating the deviation of the speech speed range.

[0037] As a further solution, in step G1, the multi-dimensional intelligent arrangement of courses is carried out through the following steps:

[0038] Segment the course video after noise reduction to obtain a plurality of first video segments, and set a marker A for each first video segment; wherein the initial value of the marker A is 0;

[0039] Use YOLO to detect the teacher's image area and determine whether the teacher is writing on the blackboard; if so, set the marker A of the current video clip to 1;

[0040] Determine whether there is voice in the video clip according to the voice recognition result; if so, set the marker A of the current video clip to 1;

[0041] The first video segment with marker A=1 is retained, the first video segment with marker A=0 is discarded, and the video segments are synthesized according to the timeline to obtain the corresponding organized course video.

[0042] As a further solution, when there are gaps between video clips, several key frames of the previous and next video clips are extracted respectively, and the key frames are arranged in timeline order. Super SloMo is used as an interpolation model, and several key frames of the previous and next video clips are input to generate intermediate frames to obtain transition pictures. The number of intermediate frames generated is determined according to the MOOC playback frame rate, and the transition pictures are filled with silent audio or preset transition picture audio.

[0043] As a further solution, in step G4, the synthesis process is performed by the following steps:

[0044] Match the edited course video and the synthesized new voice by segments, and segment them into new voice segments and second video segments;

[0045] Obtain the length of the audio track corresponding to the new voice segment, and compare it with the video length of the second video segment;

[0046] If the length of the audio track is greater than the length of the video, a frame insertion operation is performed;

[0047] If the length of the audio track is shorter than the length of the video, a frame extraction operation is performed;

[0048] If the length of the audio track is equal to the length of the video, the speed alignment of the current matching segment is completed;

[0049] Obtain the translated text corresponding to the current language and / or the translated text corresponding to the language selected by the user;

[0050] Adding subtitles to the second video segment after the speed change alignment is completed; wherein the subtitles come from the translated text of the new voice segment matching the second video segment;

[0051] Complete the speed alignment and subtitle addition of all new voice clips and the second video clips, and synthesize the finished video translated into the corresponding language.

[0052] As a further solution, the frame insertion operation: inserting generated frames into the second video segment and making the video length equal to the audio track length; the frame extraction operation: extracting picture frames from the second video segment and making the video length equal to the audio track length; wherein, if there is a silent segment, the silent segment is included in the previous segment for matching and speed alignment.

[0053] On the other hand, the present invention provides a system for automatically generating multilingual MOOCs based on recorded courses, including an intelligent noise reduction module, an intelligent speech recognition module, an intelligent speech recognition result correction module, an intelligent voiceprint training sample extraction module, a voiceprint model training module and a multilingual video synthesis module; wherein,

[0054] Intelligent noise reduction module: Provides intelligent noise reduction processing services for regular recorded course video files to obtain noise-reduced course videos;

[0055] Intelligent speech recognition module: provides intelligent speech recognition services for the course videos after noise reduction and obtains the corresponding speech recognition results;

[0056] Intelligent speech recognition result correction module: provides intelligent speech recognition result correction service for speech recognition results to obtain corresponding corrected speech recognition results;

[0057] Intelligent voiceprint training sample extraction module: provides intelligent voiceprint training sample extraction service for the corrected speech recognition results to obtain the corresponding voiceprint sample files;

[0058] Voiceprint model training module: provides voiceprint model training services for voiceprint sample files. After the training is completed, the teacher's voiceprint model is obtained and the corresponding model ID is assigned;

[0059] Multilingual video synthesis module: calls the corresponding teacher's voiceprint model through the model ID and generates a finished video; it includes a multilingual translation module, a TTS multilingual synthesis module, and a synthesis module;

[0060] Multilingual translation module: provides multilingual translation services for the sorted speech recognition results to obtain the corresponding translated text;

[0061] TTS multi-language synthesis module: provides TTS multi-language synthesis for the translated text and the teacher's voiceprint model to obtain the synthesized new voice;

[0062] Synthesis module: Provide synthesis processing for the synthesized new voice and the sorted course video to obtain the finished video translated into the corresponding language; the synthesis processing includes speed alignment and subtitle addition.

[0063] Compared with the related art, the method and system for automatically generating multilingual MOOCs based on recorded courses provided by the present invention have the following advantages:

[0064] 1. The present invention innovatively proposes a method for automatically converting routine recorded course videos into MOOC videos in different languages ​​in batches without manual intervention;

[0065] 2. The present invention selects and designs five core feature sub-items: phoneme diversity, emotional stability, sentence length, background noise and speaking speed, and introduces the uniqueness of adaptive weight dynamic adjustment based on feature distribution, feature sub-item calculation method and comprehensive scoring mechanism; it can effectively solve the data screening problem in the automatic extraction of voiceprint training in recorded courses, and enhance the robustness of the voiceprint model;

[0066] 3. The present invention innovatively proposes a multi-dimensional intelligent arrangement service for courses, which combines voice, text and behavior to correct each other, solves the problem of correcting voice recognition in normal classes and the problem of removing unimportant content, and realizes the refined arrangement of content when converting courses into exquisite course videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of the steps of a method for automatically generating multilingual MOOCs based on recorded courses provided by the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0069] See also Figure 1 The present invention provides a method for automatically generating multilingual MOOCs based on recorded courses, and performs PPT alignment for live teaching through the following steps:

[0070] Step A: Collect the regular recorded course video files of the current recorded course;

[0071] Step B: Perform intelligent noise reduction processing on the regular recorded course video files to obtain the noise-reduced course videos;

[0072] Step C: Perform intelligent speech recognition on the course video after noise reduction to obtain the corresponding speech recognition result;

[0073] Step D: performing intelligent speech recognition result correction on the speech recognition result to obtain a corresponding corrected speech recognition result;

[0074] Step E: Extract intelligent voiceprint training samples from the corrected speech recognition results to obtain corresponding voiceprint sample files;

[0075] Step F: Train the voiceprint model using the voiceprint sample file. After the training is completed, the teacher's voiceprint model is obtained and assigned a corresponding model ID;

[0076] Step G1: Perform multi-dimensional intelligent sorting of the course video after noise reduction and the corrected speech recognition result to obtain sorted course video and sorted speech recognition result;

[0077] Step G2: translating the sorted speech recognition results into multiple languages ​​to obtain corresponding translated texts;

[0078] Step G3: Perform TTS multi-language synthesis based on the translated text and the teacher's voiceprint model to obtain a synthesized new voice;

[0079] Step G4: The synthesized new voice and the sorted course video are synthesized to obtain a finished video translated into the corresponding language; wherein the synthesis process includes speed alignment and subtitle addition.

[0080] It should be noted that the purpose of the present invention is to provide a method for batch-automatically converting the normal recorded course videos into MOOC videos in different languages. Without manual intervention, the accuracy of the speech recognition results based on the course content can be improved. It innovatively proposes how to automatically extract voiceprint training samples based on the teacher's voice in class using speech recognition, speech speed and phoneme diversity judgment, and solves the problem of how to automatically extract voiceprint samples that meet the requirements of voiceprint model training. Combined with the multi-dimensional intelligent arrangement service of the course, combined with behavioral analysis and comprehensive judgment of multiple dimensions such as stop words, the problem of removing some unimportant content in the normalized class process is solved, the content is finely arranged, and combined with more mature translation, speech synthesis and automatic scaling and speed alignment methods, the automatic generation of new course videos is realized, so that batch translation and conversion processing can be performed on a large number of recorded courses in colleges and universities, while maintaining the teacher's own voice characteristics and class content. For the current professional recording and post-production methods, a lot of manpower and material resources are saved, and rapid and large-scale internationalization of courses can be achieved.

[0081] In step A, a course recorded by a regular recording host is obtained. The recording host has a fixed position and can obtain the course video recorded by the teacher. The recorded course video completely captures the PPT screen area, the blackboard area and the teacher's image.

[0082] In step D, OCR recognition is performed on the PPT screen area of ​​the video file to obtain the OCR-recognized text. Important nouns, proper nouns, etc. are extracted from the text through the NLP analysis service to create a PPT keyword library for the course. Regular expressions are used to detect typos or commonly misrecognized words in speech recognition, and they are automatically replaced with terms in the PPT keyword library, thereby completing automatic correction of speech recognition results.

[0083] In step E, the content processed during the speech period is automatically extracted based on the voiceprint training samples determined by voice, speech rate, emotion, and phoneme diversity, avoiding the inclusion of spoken language, selecting sentences with rich phonemes at too fast or too slow speech rates, and ensuring that the voiceprint model can learn complete acoustic features. The most suitable learning scenario here is the addition of emotional stability scores. When teaching, teachers will use emotional tone to assist in expressing the content, so that students can focus on the key points, concentrate their attention, and assist in expression.

[0084] In a specific embodiment:

[0085] Item 1: Phoneme Diversity Score P

[0086] How it works: We measure phonemic diversity by calculating the proportion of unique phonemes in a sentence.

[0087] Extract phoneme sequences.

[0088] Count the number of unique phonemes and the total number of phonemes.

[0089] Calculating phoneme diversity scores

[0090] Example: In the sentence "Calculus is the science of change", the phoneme sequence is [w, ei, j, i, b, f, a, k, x, u, e, n, g], the number of unique phonemes is 11, the total number of phonemes is 15, and the score is:

[0091] Item 2: Set the sub-item emotional stability score E, which is mainly used to select sentences with rich phonemes to cover as many pronunciation features as possible and ensure that the voiceprint model can learn complete acoustic features.

[0092] Implementation method: Calculate emotional stability based on the standard deviation of emotional intensity fluctuations.

[0093] Divide the sentence into smaller speech segments based on words.

[0094] The sentiment analysis model BERT is used to calculate the sentiment intensity value of each segment.

[0095] Calculate the emotional fluctuation (standard deviation) V, scoring formula: Where K is the amplification factor, which is used to adjust the score impact.

[0096] Example: In the sentence "Calculus is the science of change", the sentiment intensity values ​​are [0.62, 0.64, 0.66, 0.65, 0.63], the mean is 0.64, and the standard deviation V=0.014.

[0097] Item 3: Sentence length score L

[0098] Implementation: Score based on whether the sentence length is within the ideal range

[0099] Set the ideal length range to (Generally speaking, 10 to 15 words)

[0100] If the sentence length T is within the ideal range, full marks will be awarded. Otherwise, marks will be deducted according to the deviation.

[0101]

[0102] Example: The sentence "Calculus is the science of change" has a length of 10, which meets the ideal length range and has a score of L = 100

[0103] Example: Sentence "Calculus helps us analyze and understand the relationship between the rate of change and the cumulative amount in a continuous process.", length 18, L=-20

[0104] Item 4: Background noise score N

[0105] Implementation method: The impact of background noise on sentence quality is evaluated based on the signal-to-noise ratio (SNR).

[0106] Use Librosa to calculate the SNR value of the audio.

[0107] According to the SNR score formula:

[0108] ,in

[0109] Example: The audio corresponding to the sentence "Calculus is the science of change" first uses voice activity detection VAD to analyze whether the audio has obvious noise segments and intervals, and then uses Python and Librosa library to perform SNR. After the actual audio test, the SNR is calculated = 25

[0110] Item 5: Set the speaking speed score item R to select sentences with even speaking speed and clear pronunciation, and avoid speaking too fast or too slow.

[0111] Implementation method: Calculate the score based on whether the speaking speed is moderate.

[0112] The speed S of the sentence is measured as words per minute, in sentences.

[0113]

[0114] Set the ideal speaking speed range The commonly recommended speaking speed is 120~160 words / minute. Calculate the score:

[0115]

[0116] Adaptive weight dynamic adjustment based on feature distribution

[0117] We can set basic weights for each sub-feature and dynamically adjust the weights according to their distribution standard deviations, making the scoring mechanism more intelligent and adaptable, so as to make the scoring system more intelligent and highly customized. Specifically, based on the dynamic adjustment of adaptive weights based on feature distribution, we hope to give higher weights to features with larger standard deviations in the sample feature distribution, and give lower weights to features with smaller standard deviations, i.e., concentrated distribution. This can avoid scoring deviations caused by the concentrated performance of a certain feature.

[0118] Weight adjustment formula

[0119] The base weight is set to That is, the fixed weight is set according to the feature importance, and the dynamic weight is set according to the feature standard deviation Make adjustments:

[0120]

[0121] in Represents the standard deviation of feature i, indicating the distribution of this feature in the scoring sample

[0122] Represents the sum of the standard deviations of all features, used to standardize the weights of each feature

[0123] Complete scoring formula:

[0124] When calculating the final score S, we divide each feature score into With its adaptive weight Multiply then add:

[0125]

[0126] in:

[0127] Indicates the score of each feature, such as phoneme diversity score, emotional stability score, etc. It is the adaptively adjusted weight, which is dynamically adjusted based on the feature distribution.

[0128] Example scoring comparison of step D:

[0129] Suppose we have the following two common calculus course statements:

[0130] Statement A: "Calculus is the science of change."

[0131] Statement B: "Calculus helps us understand the relationship between rate of change and amount of accumulation."

[0132] Assuming feature distribution:

[0133] Phoneme diversity standard deviation

[0134] Emotional stability standard deviation

[0135] Sentence length standard deviation

[0136] Background noise standard deviation

[0137] Standard deviation of speech rate

[0138] The base weights are:

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] Dynamic weight calculation:

[0145]

[0146] Dynamic weights of each feature:

[0147] Phoneme diversity 0.04, emotional stability 0.075, sentence length 0.015, background noise 0.048, speaking speed 0.032

[0148] The feature calculation scores of sentences A and B are shown in Table 1:

[0149] Table 1 Feature calculation score table

[0150]

[0151] Final calculation:

[0152] =18.6082, =19.2625

[0153] In step G1, the core function of the course multi-dimensional intelligent arrangement service is that the recorded course is the teacher's class video, which needs to realize automatic course conversion and needs to correct and arrange the content of the class based on multi-dimensional information:

[0154] First, the teacher's voice recognition result may be wrong, and we hope to correct it based on the OCR recognition result in the PPT screen;

[0155] Secondly, the recorded course is a complete record of the entire teaching process, but when we use it as raw material to convert it into a high-quality course, some content fragments do not need to be retained. We hope to be able to determine whether the content of the screen should be retained by judging whether there is human voice and the teacher's behavior. The processing steps are as follows:

[0156] Speech recognition results: Get the speech recognition results of the course video obtained in the previous process, perform text segmentation on the results, and use TF-IDF to convert them into vector representations to generate the text results R;

[0157] Prepare OCR results for inclusion in the subject word library: If OCR is used to recognize the entire picture, a lot of messy text will be introduced due to the edges. In the recorded course, the positions of the PPT screen area and the blackboard area are often fixed. Therefore, YOLO is first used to recognize the area of ​​the entire picture, locate the PPT area, and only perform OCR recognition on the picture content in the PPT area to extract the text and generate structured text data. The recognized text data is segmented and converted into a vector representation using TF-IDF to extract the core words, create a subject word library for this video, and add the extracted core words to the subject word library P;

[0158] The results of speech recognition are corrected by the content of the screen, and the results of speech recognition are compared with the content in the subject word library through the cosine similarity algorithm:

[0159]

[0160] If the similarity is lower than the set threshold , then make correction

[0161] =

[0162] The behavior and audio are combined to correct the video content: YOLO is used to detect whether the teacher is writing on the blackboard. The results of behavior detection are combined with the results of speech recognition to generate the teacher's behavior score A. If the video clip has speech recognition results, A=1. If there is no speech recognition result, but the behavior recognizes that the teacher is writing on the blackboard, then A=1 is set. If the video clip has no speech recognition results and no blackboard writing behavior, then A=0. During synthesis, only video clips with A=1 are retained.

[0163] Video clip splicing process for actions after removing segments without speech recognition results and without blackboard writing behavior: Since video segments are removed, if directly spliced, there may be a jumpy discomfort visually. Therefore, in order to keep the transition between the two segments of the final video smooth and without an abrupt frame skipping effect visually, an interpolation service for generating intermediate transition frames is performed here by calculating the optical flow of the front and rear frames to predict the position and pose of the intermediate frame. First, extract the key frames before and after the splicing point: Frames need to be inserted at the splicing point of the two segments. Extract 5 frames before and after the splicing point to ensure sufficient key frame information on both sides. Select Super SloMo as the interpolation model and input the front and rear frames to generate intermediate frames. The number of generated frames is determined by the frame rate. If the source video is 25fps, 30 frames can be generated in 1 second to facilitate a more natural transition at the splicing point. The audio corresponding to the 1-second transition video generated is filled with silence.

[0164] Sound correction based on the speech recognition result: Compare the text of the speech recognition result with the stop list of oral words (such as "um", "uh", "ah"). Considering that oral words usually appear at the beginning of a sentence or at pauses in the middle of a sentence, remove the oral words in the text of the speech recognition result. In this way, such invalid words will not exist in the new speech generated using the teacher's trained voiceprint.

[0165] In step G4, the synthesis service: Initiate a translation service for the speech recognition result corresponding to the sorted course video in units of sentences to obtain the translated text of the corresponding sentence. Initiate the speech generation of the teacher's voiceprint model with the text to obtain the new speech Ae of the sentence. Perform frame filling or frame extraction operations on the video of the original sentence according to the length of Ae. When the length of Ae is greater than the length of the video of the original sentence, perform frame filling operations on the video of the original sentence. When the length of Ae is less than the length of the video of the original sentence, perform frame extraction operations on the video of the original sentence. Finally, adjust the video length to be consistent with the length of the new speech Ae. If there are silent segments, include the silent segments in the previous sentence for variable speed alignment processing. At the same time, display the original text and the translated text of this sentence on the video screen. After all sentences are processed, a new course video with the translation of the corresponding language of the video is obtained.

[0166] The above embodiments only represent relatively preferred implementation manners, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the invention, several modifications and improvements can be made, and these all belong to the protection scope of the invention.

Claims

1. A method for automatically generating multilingual MOOCs based on recorded courses, characterized in that: Perform the following steps to align the live teaching PPT: Step A: Collect the regular recorded course video files of the current recorded course; Step B: Perform intelligent noise reduction processing on the regular recorded course video files to obtain the noise-reduced course videos; Step C: Perform intelligent speech recognition on the course video after noise reduction to obtain the corresponding speech recognition result; Step D: performing intelligent speech recognition result correction on the speech recognition result to obtain a corresponding corrected speech recognition result; Step E: Extract intelligent voiceprint training samples from the corrected speech recognition results to obtain corresponding voiceprint sample files; Step F: Train the voiceprint model using the voiceprint sample file. After the training is completed, the teacher's voiceprint model is obtained and assigned a corresponding model ID; Step G1: Perform multi-dimensional intelligent sorting of the course video after noise reduction and the corrected speech recognition result to obtain sorted course video and sorted speech recognition result; Step G2: translating the sorted speech recognition results into multiple languages ​​to obtain corresponding translated texts; Step G3: Perform TTS multi-language synthesis based on the translated text and the teacher's voiceprint model to obtain a synthesized new voice; Step G4: synthesizing the synthesized new voice and the sorted course video to obtain a finished video translated into the corresponding language; wherein the synthesis process includes speed alignment and subtitle addition; In step E, intelligent voiceprint training sample extraction is performed through the following steps: Sentence segmentation is performed on the corrected speech recognition result to obtain a number of sentence units and the saliva words are filtered out through the stop saliva word library; Perform phoneme diversity scoring on the sentence unit to obtain the phoneme diversity score of the corresponding sentence unit; Score the emotional stability of the sentence unit to obtain the emotional stability score of the corresponding sentence unit; Sentence length scoring is performed on the sentence unit to obtain the sentence length score of the corresponding sentence unit; Scoring the background noise of the sentence unit to obtain the background noise score of the corresponding sentence unit; Score the speech speed of the sentence unit to obtain the speech speed score of the corresponding sentence unit; Based on the phoneme diversity score, emotional stability score, sentence length score, background noise score and speech speed score, each sentence unit is comprehensively scored to obtain a comprehensive score of the sentence unit; the comprehensive score adopts a comprehensive scoring mechanism with adaptive weights; The sentence units are sorted according to the comprehensive scores, and sentence units with a set ratio are selected from high to low, spliced ​​into a complete audio file, and used as a voiceprint sample file.

2. A method for automatically generating multilingual MOOCs based on recorded courses according to claim 1, characterized in that: In step A, image regions are extracted from the regular recorded course video files; the image regions include the PPT screen region, the blackboard region and the teacher image region.

3. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 2, characterized in that: In step D, the intelligent speech recognition result correction is performed through the following steps: Perform OCR recognition on the PPT screen area of ​​the video file to obtain OCR recognized text; Through NLP analysis of OCR recognized text, correction terms are extracted and a PPT keyword library corresponding to the current recorded course is created; correction terms include important nouns, proper nouns and high-frequency nouns; Regular expressions detect typos or commonly misrecognized words in speech recognition results, and compare similarities with correction terms in the PPT keyword library; If the similarity reaches the correction judgment threshold, the corresponding typos or commonly misrecognized words are automatically replaced with correction terms in the PPT keyword library.

4. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 3, characterized in that: The sentiment stability score is calculated by the following steps: the sentence unit is divided into several speech segments based on words; the sentiment analysis model BERT is used to calculate the sentiment intensity value of each speech segment; Calculate the standard deviation V of emotional fluctuations; Calculate the emotional stability score through the emotional score formula ;in, , K It is the magnification factor used to adjust the score impact; The phoneme diversity score is performed by the following steps: extracting the phoneme sequence of the sentence unit; identifying the number of unique phonemes and the total number of phonemes in the phoneme sequence; calculating the phoneme diversity score ;in, ； Sentence length score: obtained by setting the ideal length range and calculating the deviation of the sentence length range; Background noise score: evaluated by the signal-to-noise ratio (SNR) to assess the impact of background noise on sentence quality; Speech speed score: obtained by setting the ideal speech speed range and calculating the deviation of the speech speed range.

5. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 1, characterized in that: In step G1, the multi-dimensional intelligent arrangement of courses is carried out through the following steps: Segment the course video after noise reduction to obtain a plurality of first video segments, and set a marker A for each first video segment; wherein the initial value of the marker A is 0; Use YOLO to detect the teacher's image area and determine whether the teacher is writing on the blackboard; if so, set the marker A of the current video clip to 1; Determine whether there is voice in the video clip according to the voice recognition result; if so, set the marker A of the current video clip to 1; The first video segment with marker A=1 is retained, the first video segment with marker A=0 is discarded, and the video segments are synthesized according to the timeline to obtain the corresponding organized course video.

6. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 1, characterized in that: When there are gaps between video clips, several key frames of the previous and next video clips are extracted respectively, and the key frames are arranged in timeline order. Super SloMo is used as the interpolation model, and several key frames of the previous and next video clips are input to generate intermediate frames to obtain transition pictures. The number of intermediate frames generated is determined according to the MOOC playback frame rate, and the transition pictures are filled with silent audio or preset transition picture audio.

7. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 1, characterized in that: In step G4, the synthesis process is performed through the following steps: Match the edited course video and the synthesized new voice by segments, and segment them into new voice segments and second video segments; Obtain the length of the audio track corresponding to the new voice segment, and compare it with the video length of the second video segment; If the length of the audio track is greater than the length of the video, a frame insertion operation is performed on the second video segment; If the length of the audio track is less than the length of the video, a frame extraction operation is performed on the second video segment; If the length of the audio track is equal to the length of the video, the speed alignment of the current matching segment is completed; Obtain the translated text corresponding to the current language and / or the translated text corresponding to the language selected by the user; Adding subtitles to the second video segment after the speed change alignment is completed; wherein the subtitles come from the translated text of the new voice segment matching the second video segment; Complete the speed alignment and subtitle addition of all new voice clips and the second video clips, and synthesize the finished video translated into the corresponding language.

8. The method for automatically generating multilingual MOOCs based on recorded courses according to claim 7, characterized in that: Frame insertion operation: insert generated frames into the second video segment, and make the video length equal to the audio track length; frame extraction operation: extract picture frames from the second video segment, and make the video length equal to the audio track length; among them, if there is a silent segment, the silent segment will be included in the previous segment for matching and speed alignment.

9. A system for automatically generating multilingual MOOCs based on recorded courses, characterized in that: It includes intelligent noise reduction module, intelligent speech recognition module, intelligent speech recognition result correction module, intelligent voiceprint training sample extraction module, voiceprint model training module and multilingual video synthesis module; among them, Intelligent noise reduction module: provides intelligent noise reduction processing services for regular recorded course video files to obtain the noise-reduced course videos; Intelligent speech recognition module: provides intelligent speech recognition services for the course videos after noise reduction and obtains the corresponding speech recognition results; Intelligent speech recognition result correction module: provides intelligent speech recognition result correction service for speech recognition results to obtain corresponding corrected speech recognition results; Intelligent voiceprint training sample extraction module: provides intelligent voiceprint training sample extraction service for the corrected speech recognition results to obtain the corresponding voiceprint sample files; Voiceprint model training module: provides voiceprint model training services for voiceprint sample files. After the training is completed, the teacher's voiceprint model is obtained and the corresponding model ID is assigned; Multilingual video synthesis module: calls the corresponding teacher's voiceprint model through the model ID and generates a finished video; it includes a multilingual translation module, a TTS multilingual synthesis module, and a synthesis module; Multilingual translation module: provides multilingual translation services for the sorted speech recognition results to obtain the corresponding translated text; TTS multi-language synthesis module: provides TTS multi-language synthesis for the translated text and the teacher's voiceprint model to obtain the synthesized new voice; Synthesis module: Provide synthesis processing for the synthesized new voice and the sorted course video to obtain the finished video translated into the corresponding language; the synthesis processing includes speed alignment and subtitles; In the intelligent voiceprint training sample extraction module, intelligent voiceprint training sample extraction is performed through the following steps: Sentence segmentation is performed on the corrected speech recognition result to obtain a number of sentence units and the saliva words are filtered out through the stop saliva word library; Perform phoneme diversity scoring on the sentence unit to obtain the phoneme diversity score of the corresponding sentence unit; Score the emotional stability of the sentence unit to obtain the emotional stability score of the corresponding sentence unit; Sentence length scoring is performed on the sentence unit to obtain the sentence length score of the corresponding sentence unit; Scoring the background noise of the sentence unit to obtain the background noise score of the corresponding sentence unit; Score the speech speed of the sentence unit to obtain the speech speed score of the corresponding sentence unit; Based on the phoneme diversity score, emotional stability score, sentence length score, background noise score and speech speed score, each sentence unit is comprehensively scored to obtain a comprehensive score of the sentence unit; the comprehensive score adopts a comprehensive scoring mechanism with adaptive weights; The sentence units are sorted according to the comprehensive scores, and sentence units with a set ratio are selected from high to low, spliced ​​into a complete audio file, and used as a voiceprint sample file.

Images