Method for assisting multi-mode speech recognition based on slide information in classroom scene

Abstract

The invention relates to a method for assisting multi-mode speech recognition based on slide information in a classroom scene, and belongs to the field of speech recognition tasks. The method comprises the following steps: firstly, formulating basic information including subject names, online courses, learning segments, grades and the like of subject-related keywords to construct accurate search conditions; then, through a video search module, the system performs video retrieval in the platform resources to obtain target education video resources; then, the video is downloaded and preprocessed, including slide text extraction and candidate fragment production, and the data is cleaned and formatted to ensure high quality of the data; and finally, generating corpus system data from the preprocessed text and audio data, and identifying the audio data by adopting a voice identification model. According to the invention, by combining the slide content and the voice recognition technology, the voice transcription precision in a complex scene is effectively improved, and the method has important education application value.

Description

Technical Field

[0001] The present invention relates to a method for assisting multimodal speech recognition based on slide information in a classroom scenario, and belongs to the field of speech recognition tasks. Background Art

[0002] The application of speech recognition related technologies in classroom scenarios has far-reaching significance. The recognition results can help generate text records of classroom audio, which is convenient for students to review and teachers to review. However, due to the particularity of offline classroom environments, such as background noise, multiple speakers, long recording distances, and complex and specific teaching vocabulary, traditional speech recognition systems face huge challenges in such specific scenarios. The present invention aims to improve the effectiveness of speech recognition systems through slides in classroom teaching. In teaching scenarios, auxiliary information usually includes visual resources related to the teaching content, such as slides or writing on the blackboard. The text information contained in these visual materials (such as keywords and important noun concepts) can provide additional contextual support for speech recognition, enabling the system to more accurately identify specific voice segments, thereby improving recognition accuracy. This invention proposes a method for constructing a teacher's classroom corpus using slides to assist speech recognition tasks. This method integrates the text information in the slides and uses the Funasr framework to improve the accuracy of classroom teaching speech recognition. The encoder part is responsible for extracting the features of the text content of classroom slides, while the decoder uses the attention mechanism to focus on processing the text information most relevant to the current voice segment, thereby improving the accuracy and robustness of recognition. This is expected to significantly improve the speech recognition effect in teaching scenarios and bring substantial progress to the field of educational technology. Summary of the invention

[0003] The technical problem to be solved by the present invention is to use additional information to assist classroom teaching speech recognition tasks in specific scenarios, so as to solve the problem of poor teacher speech recognition results in existing teaching scenarios.

[0004] The technical solution of the present invention is: a method for multimodal speech recognition assisted by slide information in a classroom scenario. This method can perform speech recognition tasks on teacher targets in teaching scenarios according to the speaking characteristics of different teachers. Compared with traditional speech recognition methods that mainly rely on audio information to perform speech recognition on teachers, the present invention takes into account the influence of external interference factors such as noise level, environmental echo, and insufficient capture quality of multimedia equipment, and inputs the acoustic features of the speaking teacher into the encoder to meet the requirements of speech recognition task input. The key to the present invention is to utilize the auxiliary information in the slides, and at the same time process the association between the teacher target speech features and the corresponding text features of the audio based on the encoder and decoder model, thereby realizing the speech recognition task of the teacher target. This method can more effectively recognize the teaching voice of classroom teachers, providing strong support for downstream tasks. The specific steps are:

[0005] Step 1: Corpus data collection and construction phase

[0006] Step 1.1: The data sources of the corpus include the Chinese University Massive Open Online Courses (MOOC) platform, the National Primary and Secondary School Smart Education Platform, and the recorded videos of offline real classes in some universities. Through diverse data sources, it covers at least 6 disciplines and more than 400 classroom teaching videos of teachers in primary and secondary schools and universities.

[0007] Step 1.2: Data screening. For the uploaded video content, screen videos that meet the research needs, and give priority to courses that contain lecture slides and the slide duration accounts for more than 65% to ensure that the selected videos have sufficient visual information to provide effective support for multimodal speech recognition.

[0008] Step 1.3: Data annotation. Generate detailed metadata for each teaching video, including the original video file (.MP4), video list ID, tags, and timestamps. Each clip is annotated with the corresponding course source information and corresponding text content, providing a rich data foundation for subsequent processing.

[0009] Step 1.4: Audio preprocessing: Convert the audio part of the video into a 16k sampling rate, single channel, 16-bit signed integer format, and slice the audio for subsequent speech recognition and analysis.

[0010] Step 2: Dataset construction and hot word extraction stage

[0011] Step 2.1: Training set and test set division. According to the data fragments extracted from the corpus, they are randomly divided into training set and test set with a ratio of 4:1 to ensure the independence and scientificity of the data sets. The training set contains 70,715 data and the test set contains 17,451 data.

[0012] Step 2.2: Generate a hot word list. Apply optical character recognition (OCR) technology to each video clip to extract the text content in the slides, generate a long text, and use keyword technology to extract keywords from the text to form a hot word list. The hot word list includes entity nouns, place names, and specialized terms in related courses, providing guidance for semantic matching in speech recognition.

[0013] Step 2.3: Data consistency check. Verify the segments in the training set and the test set, randomly sample 100 segments for comparison, and ensure that more than 96% of the segments in the data set contain the corresponding lecture slide information, thereby ensuring the consistency and integrity of the data.

[0014] Step 2.4: Data deduplication and optimization. Ensure that the training set and the test set do not overlap in categories and IDs, exclude duplicate data, and prioritize the teaching videos of liberal arts courses based on the research objectives.

[0015] Step 3: Data cleaning and processing

[0016] Step 3.1: Audio data denoising: Use audio processing tools to perform preliminary denoising on the audio data in the corpus to reduce background noise and interference, so as to improve the accuracy of subsequent speech recognition.

[0017] Step 3.2: Audio segment screening. Use voice activity detection technology (VAD) to screen the length of audio segments, remove segments that are too short (such as about 1 second) or contain catchphrases such as "um" and "uh", as well as audio with strong dialects and accents, to ensure data quality.

[0018] Step 3.3: Unify audio features. Perform regularization on the audio clips, including adjusting the speech rate, equalizing the audio loudness, eliminating differences between clips, and ensuring that all audio data has consistent features.

[0019] Step 3.4: Text annotation correction. Match and proofread the text generated by OCR recognition and automatic speech recognition system (ASR) to ensure the relevance of text content and audio content and optimize the data quality of the corpus.

[0020] Step 4: Corpus output and application stage

[0021] Step 4.1: Format the corpus output. The processed audio and its corresponding text transcription results, hot word list, and metadata information are sorted and output to generate a standardized corpus file for easy calling and checking in subsequent experiments.

[0022] Step 4.2: Multimodal speech recognition model training. Based on the corpus data, the multimodal speech recognition model is trained. By combining the hot word list and slide information, the model's recognition accuracy for speech in classroom scenarios is improved.

[0023] Step 4.3: Application testing and optimization. Test the model in multiple speech recognition tasks to verify its performance in complex environments of different disciplines, and continuously optimize the corpus based on the test results.

[0024] Step 4.4: Corpus update and maintenance. Regularly update the corpus based on newly acquired course videos and voice data to ensure that the corpus always contains the latest classroom teaching data to maintain its practicality and foresight in education research.

[0025] The present invention uses classroom audio acoustic features as one of the input conditions, and combines the hot words obtained from the slide information as an auxiliary modality to apply to the classroom speech recognition task, and constructs a corpus about Chinese teachers' classroom teaching. Compared with traditional speech recognition methods, speech recognition is performed in combination with the text content of classroom slides contained in the teaching scene. When using traditional speech recognition methods to perform speech recognition on the target speech content of teachers in scenarios interfered by external factors, there is a great problem of low recognition accuracy. In this method, additional text information is used as a supplement, so that the final recognition effect is accurate, and the accuracy of the recognition results is significantly improved.

[0026] The beneficial effects of the present invention are: combining the acoustic characteristics of the speaking teacher and the classroom slide content to assist the speech recognition task, achieving accurate speech recognition in the teacher's classroom scene, meeting user needs while improving the accuracy of the speech recognition results, better extracting the acoustic characteristics of the audio content, and improving the feature fusion effect of the audio and the corresponding text labels, making the speech recognition results clearer and more accurate. In addition, the present invention has a wide range of application value and is suitable for specific scenarios, such as traditional classrooms, online education platforms, lectures, vocational training, and online tutoring. This technological innovation is expected to play an important role in the fields of educational technology, health care, speech recognition, etc., and has great research significance and commercial potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the present invention;

[0028] Figure 2 is the benchmark pipeline graph for the corpus;

[0029] Figure 3 It is the qualitative result of the data set of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Example 1: Figure 1 As shown, a method for assisting multimodal speech recognition based on slide information in a classroom scenario, the specific steps are:

[0032] Step 1: Candidate video screening stage

[0033] Step 1.1: Data source selection

[0034] The selected corpus data sources include well-known online course platforms (MOOCs and high-quality primary and secondary school courses) and offline classroom videos of some universities. The video types cover courses in multiple disciplines such as science, liberal arts, and business, with different grades and teaching modes, suitable for various scenarios such as ordinary classrooms, discussion classrooms, and experimental classrooms. The selected data sources meet the needs of speech recognition research in terms of content coverage, video quality, and subject diversity, providing a data basis for subsequent in-depth analysis.

[0035] Step 1.2: Design subject keywords

[0036] We designed a keyword set based on subject names, course topics, grade information, and other factors to ensure efficient and accurate screening of video resources that meet research needs. Through preliminary video screening, we optimized the keyword generation mechanism using a specific subject noun set, and considered the commonly used multi-level classification tags in the corpus during screening to cover more comprehensive and extensive data, while reducing interference from irrelevant content and improving data retrieval efficiency.

[0037] Step 1.3: Manually screen video content

[0038] The video samples obtained in the initial screening were screened to ensure that the slide content and presentation rhythm of each video met the experimental requirements. The screening criteria included: the relevance of the slide content to the research objectives, the slide display time in the video should exceed 65% of the total duration and meet the subject requirements. The process ensures rich visual content, thereby providing sufficient contextual information assistance in the subsequent speech recognition task and improving the recognition accuracy of the overall system.

[0039] Step 1.4: Get video data in batches

[0040] Use a dedicated download tool to batch download videos that meet the requirements, ensuring the high-definition quality and content integrity of each video. During the download process, manage files of different resolutions, give priority to videos with a quality of 1080p and above, and save all videos in the standard MP4 format.

[0041] Step 2: Hot word list generation stage

[0042] Step 2.1: Extraction of intermediate frames from candidate videos

[0043] Through frame sampling technology, frame images are randomly intercepted from candidate videos, and frames that can contain the slide content to the greatest extent are selected for different scenes. OCR technology is used to extract text information in the image, and the text information is used as the basis for subsequent semantic analysis.

[0044] Step 2.2: Text standardization and formatting

[0045] Process the text extracted by OCR, remove irrelevant characters and background noise (such as meaningless punctuation and watermark information), and unify the font, font size and punctuation. Convert the processed text into a structured data format (JSON) so that it can be used directly in subsequent steps.

[0046] Step 2.3: Keyword extraction and hot word list generation

[0047] The extracted text content is semantically analyzed to extract keywords that are highly relevant to the course content, and high-frequency words are screened through frequency analysis and high-frequency methods. The obtained keyword set is used to generate a hot word list, which is used as a keyword matching guide for subsequent speech recognition. At the same time, the system's recognition efficiency of key content is improved by cross-matching hot words with video clips.

[0048] Specifically, the hot word list includes place names, personal names, polyphones and proprietary terms. By adding the hot word list, auxiliary modalities such as lip reading can be used to carry out speech recognition tasks, and when such semantic entity words are recognized, the accuracy of speech recognition is effectively enhanced.

[0049] Step 3: Candidate fragment generation stage

[0050] Step 3.1: Audio processing and segmentation

[0051] Use tools and voice activity detection technology (VAD) to segment audio files, remove silent segments without speech, and retain high-quality audio segments in each video. Based on factors such as speech speed and voice activity frequency in the audio, the audio is segmented into sub-segments of 3 to 5 minutes in length to ensure that each audio segment contains a complete sentence, improving the quality of subsequent speech recognition.

[0052] Step 3.2: Speech Recognition and Text Transcription

[0053] The segmented audio is recognized through the ASR system to generate accurate text transcriptions. The recognition results are matched and compared with the original audio clips to ensure highly accurate transcription text output.

[0054] Step 3.3: Generate candidate fragment output

[0055] The audio clip and the corresponding text transcription file are output and saved to form a structured file containing the audio clip path and text. The output format is structured data JSON format.

[0056] Step 4: Data sorting and cleaning

[0057] Step 4.1: Standardize the audio format. The encoding format, sampling rate, bit rate, etc. of all audio files are uniformly processed, and the audio files are converted into a standard channel mode, WAV format, to meet the format requirements of the speech recognition system.

[0058] Step 4.2: Voice segment screening

[0059] The generated speech segments are screened again to remove segments with a duration of less than 3 seconds and segments with dialect accents to ensure the language consistency and clarity of the data. For audio segments with obvious noise interference, enhancement processing is adopted to improve the quality to ensure that the quality of the final corpus reaches high standards.

[0060] Step 4.3: Data regularization

[0061] The final filtered audio data is smoothed, including speech rate adjustment and loudness equalization. The audio regularization algorithm is applied to make the speech rate and volume conform to the optimal parameter settings of the speech recognition system. Regularized audio clips can significantly improve the stability and accuracy of recognition, providing high-quality input data for speech recognition tasks.

[0062] Figure 2 The figure shows the benchmark pipeline diagram of the corpus. The corpus contains a large number of video clips of teachers teaching, corresponding text information, OCR model to extract text information of slides, etc. For each speech segment, the middle frame image is extracted, and the text information in the slide is extracted, and then the text is reformatted to obtain formatted long text, which is extracted using keyword technology, and contextual ASR is used to enhance the recognition of specific terms to facilitate subsequent research on speech recognition.

[0063] Figure 3 Qualitative results on the dataset are shown, showing recognition for ground truth, speech only, and audio-visual combined. Note how the keywords of the slide help correct the recognition. Errors in the recognition are highlighted in red and the corrected words are highlighted in blue compared to the ground truth.

[0064] Based on the specific implementation methods, the effectiveness of the technical solution of the present invention is illustrated by experiments below:

[0065] Three models were used for training and testing in the experiment, and the effects under different configurations were compared and analyzed. The training and testing effects of the data set on different models are shown in Table 1:

[0066] Table 1. Training and testing results of the dataset on different models

[0067]

[0068] Among them, WER represents word error rate, SER represents sentence error rate, CER represents character error rate, w / oK represents no hot words, and wK represents hot words. From Table 1, we can see that on the baseline model, the WER (Word Error Rate) of the training set and the test set are 2.93% and 5.07% respectively, and the corresponding CER (Character Error Rate) is 3.38% and 5.57%, while the SER (Sentence Error Rate) reaches 19.93% and 33.18%. In the experiment of the Seaco_paraformer model, the configurations with hot words (w / K) and without hot words (w / oK) were trained and tested respectively. Without introducing hot words, the WER of the training set and the test set are 6.81% and 8.02% respectively, the CER is 7.61% and 8.83%, and the SER is 45.78% and 49.48%. After the introduction of hot words, the performance of the model was significantly improved, with the WER of the training set and the test set reduced to 5.1% and 6.42% respectively, the CER to 5.62% and 6.92% respectively, and the SER to 33.08% and 38.46%. The results show that the speech recognition accuracy of the model has been significantly improved after the introduction of hot words.

[0069] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for assisting multimodal speech recognition based on slide information in a teacher's classroom scenario, characterized by: Step 1: Select the data source, design keyword search and combine manual screening to select candidate videos, and then use professional tools to download the candidate videos; Step 2: Extract frame images from the downloaded candidate videos and convert them into text using OCR technology. Then format the text and extract keywords to generate a hot word list. Step 3: Obtain high-quality audio clips from the candidate videos through audio processing and segmentation, use an automatic speech recognition system to convert the audio clips into high-precision text transcriptions, then match the audio clips with the corresponding text transcriptions, and output data files related to the candidate clips; Step 4: Check and convert the format of the output candidate segments to screen out the audio segments that meet the length and pronunciation clarity requirements, then regularize the screened audio segments to obtain standardized audio data, and use the speech recognition model in combination with the hot word list to complete audio data recognition.

2. According to claim 1, a method for assisting multimodal speech recognition based on slide information in a teacher's classroom scenario is characterized in that: The Step 1 is specifically as follows: Step 1.1: Select video content from mainstream online course platforms and recorded videos of offline courses in colleges and universities as data sources, so that the retrieved videos cover different subjects, grades, and teaching methods; Step 1.2: Design subject search keywords such as subject name, course name, and grade information according to the characteristics of different subjects and courses; Step 1.3: First, the retrieved videos are preliminarily screened to obtain classroom videos of teachers of different subjects and courses, and the videos all contain teachers' lecture slides. Then, the videos are extracted frame by frame, and the slide frames are detected by an automated method. The number of slide frames is counted, and finally the proportion of slide frames is calculated. The teaching videos with a slide frame ratio of more than 65% are classified as candidate videos. Step 1.4: Use a professional video downloading tool to batch download the candidate videos to obtain candidate videos with clear picture quality and complete content.

3. According to claim 1, a method for assisting multimodal speech recognition based on slide information in a teacher's classroom scenario is characterized in that: The Step 2 is specifically as follows: Step 2.1: Extract frame images from each candidate video clip downloaded, retain one frame for multiple consecutive frames, and determine the number of retained frames based on the video length to obtain representative frames that meet the course requirements. Apply the OCR model to recognize text information in the slides, and extract the title, key concepts, and explanation content of the text information. Step 2.2: Standardize the extracted titles, key concepts, and explanation contents by normalizing characters, removing redundant symbols and irrelevant content, and generating structured long text data; Step 2.3: Use keyword technology to perform semantic analysis on the generated long text data, extract keywords and phrases representing the content from the long text, and generate a hot word list corresponding to the course.

4. According to claim 1, a method for assisting multimodal speech recognition based on slide information in a teacher's classroom scenario is characterized in that: The Step 3 is specifically as follows: Step 3.1: Use tools to perform audio cropping, noise reduction, and smoothing on the candidate videos. Combined with voice activity detection technology, the audio is intelligently segmented according to voice activity to obtain complete and high-quality audio clips. Step 3.2: Use an automatic speech recognition system to recognize the segmented audio clips and generate corresponding transcription text; Step 3.3: Match the audio clip with the automatic speech recognition transcription text to obtain the audio information and path format data of the corpus, which are saved in .scp and .jsonl formats respectively.

5. The method of claim 1 for assisting multimodal speech recognition based on slide information in a teacher's classroom scenario, characterized in that: The Step 4 is specifically as follows: Step 4.1: Check the sampling rate, bit depth and channel consistency of the obtained complete and high-quality audio clips, and then convert the audio format to unify the data input format and obtain an audio file that meets the standard requirements of the speech recognition system; Step 4.2: Further screen the audio clips, using voice activity detection technology to identify clips that are too short and do not meet the requirements, as well as audio clips of modal particles of about 1 second, dialects or heavy accents, and automatically screen out such clips; Step 4.3: Perform regularization processing such as audio smoothing, speech speed adjustment, and loudness equalization on the filtered audio clips to reduce the differences in audio characteristics between different clips and obtain standardized audio data. Then, combined with the audio information and path format data of the corpus and the hot word list, the speech recognition model is used to realize the recognition of the standardized audio data.

Images