Education scene speech recognition method based on subject attention mechanism

Abstract

The application discloses an education scene voice recognition method based on a subject attention mechanism, relates to the technical field of natural language processing, and comprises the following steps: constructing a voice preprocessing model, and preprocessing education scene voice data to be recognized through the voice preprocessing model. The application preprocesses the education scene voice data to be recognized through the constructed voice preprocessing model, obtains enhanced education scene voice data, further acquires semantic information of the education scene through phoneme arrangement information, converts the education scene voice data to be recognized into a language used by a listener, sorts converted vocabulary of the education scene voice through a language grammar used by the listener, adjusts phoneme arrangement information in the converted vocabulary through the semantic information of the education scene, makes the converted vocabulary contain emotions of a lecturer, and further converts a sentence into voice through a language converter, so that the voice of the lecture video contains emotions, and the listener can better understand emotions contained in the language.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a speech recognition method for educational scenarios based on subject attention mechanisms. Background Technology

[0002] The widespread availability of educational videos allows more children to access guidance from renowned educators. However, some children may be exposed to their native dialect rather than Mandarin. Without audio conversion of these videos, it can be inconvenient for people speaking different languages ​​to watch them. For instance, if an English-speaking child watches a Chinese-language educational video and the video is translated into English and sent to them, the text cannot convey the emotions within the language. Consequently, the listener cannot experience the emotional nuances of the language and cannot understand the author's feelings when creating the text. Summary of the Invention

[0003] To address the aforementioned technical issues, this paper provides a speech recognition method for educational scenarios based on subject-specific attention mechanisms. This technical solution resolves the problem mentioned in the background that when machine-translated speech is used, the learner cannot experience the emotions in the language, thus failing to understand the author's feelings when creating the text.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Speech recognition methods for educational scenarios based on subject-specific attention mechanisms include: Acquire the voice data of the educational scenario to be identified; A speech preprocessing model is constructed to preprocess the speech data of the educational scenario to be recognized, thereby obtaining the preprocessed speech data of the educational scenario. Based on a joint algorithm of short-time energy and zero-crossing rate, the pre-processed educational scenario speech data is segmented to obtain multi-group segmented educational scenario speech data. Feature extraction processing is performed on multi-group segmented speech data of educational scenarios to obtain word class information and temporal information of multiple educational scenarios; The word class information of multiple educational scenarios is analyzed and processed to obtain the semantic information of the educational scenarios; Select the language type using the language converter; The vocabulary converter performs word class matching on multiple sets of educational scenarios based on language type to obtain the converted vocabulary; By adjusting the phonemes of the converted words based on the semantic information of the educational scenario, we can obtain the speech-to-vocabulary for the educational scenario. The vocabulary for speech conversion in educational scenarios is sorted based on the tense information of the educational scenario to obtain the correct speech conversion sentences in the educational scenario.

[0005] Preferably, the step of preprocessing the educational scenario speech data to be recognized using a speech preprocessing model to obtain the preprocessed educational scenario speech data specifically includes the following steps: The speech data of the educational scenario to be identified is sampled and processed to obtain the sampled speech signal of the educational scenario; Acquire the frequency of the audio signal in the sampled educational scenario; Denoising the sampled educational speech signals; If the frequency of the sampled educational scene speech signal is greater than or equal to the set first threshold, and the frequency of the sampled educational scene speech signal is less than or equal to the set second threshold, then the sampled educational scene speech signal is set as a clean speech signal. If the frequency of the sampled educational scene speech signal is less than the first threshold or greater than the second threshold, the sampled educational scene speech signal segment is removed. Endpoint detection is performed on the clean sampled educational scene speech signal to remove non-speech components and obtain the target educational scene speech signal. The non-speech components are specifically the silent part of the clean speech signal. Normalize the speech signal of the target educational scenario to obtain the normalized speech signal of the target educational scenario; The normalized target educational scene speech signal is enhanced by using an acoustic model to obtain preprocessed educational scene speech data.

[0006] Preferably, the joint algorithm based on short-time energy and zero-crossing rate for segmenting the preprocessed educational scenario speech data to obtain multi-segmented educational scenario speech data specifically includes the following steps: The preprocessed educational scenario audio data is divided into windows to obtain multiple sets of educational scenario audio windows; The speech signal in each educational scenario's speech window is processed to obtain multiple sets of short-time energy, namely:

[0007] In the formula, E is the short-time energy; Let N be the value of the audio signal in the educational scenario at the nth sampling point; N is the length of the audio window in the educational scenario; and n is the number of samplings. The speech signal in each educational scenario speech window is processed to obtain the zero-crossing rate of the educational scenario speech signal, i.e.:

[0008] In the formula, Z is the zero-crossing rate of the voice signal in the educational scenario; N is the length of the voice window in the educational scenario; Let n be the value of the speech signal in the educational scenario at the nth sampling point; Let be the value of the speech signal in the educational scenario at the (n-1)th sampling point; n is the number of samplings; sgn() is the sign function; The zero-crossing rate of multiple sets of short-time energy and educational scenario speech signals is judged and processed. If the short-term energy is greater than the set third threshold and the zero-crossing rate of the educational scene speech signal is less than the set fourth threshold, obtain the speech segmentation position in the educational scene. If the short-time energy is less than or equal to the set third threshold and the zero-crossing rate of the educational scene voice signal is greater than or equal to the set fourth threshold, output the continuous educational scene voice signal. The pre-processed educational scenario speech data is segmented by the location of the speech segments in the educational scenario to obtain multi-group segmented educational scenario speech data.

[0009] Preferably, the step of performing feature extraction processing on multi-group segmented educational scenario speech data to obtain part-of-speech information and temporal information for multiple educational scenarios specifically includes the following steps: Obtain part-of-speech information; Using part-of-speech information as a feature, multi-group segmented speech data of educational scenarios are extracted and processed to obtain part-of-speech information of multiple educational scenarios; Using word class information from multiple educational scenarios as features, temporal analysis is performed on segmented educational scenario speech data to obtain temporal information of the educational scenarios.

[0010] Preferably, the step of analyzing and processing the word class information of multiple sets of educational scenarios to obtain the semantic information of the educational scenarios specifically includes the following steps: Establish a first vocabulary, determine the first linguistic rules based on the first vocabulary, and decompose and process the information of each word class through the first vocabulary and the first linguistic rules to obtain syllable information; Based on the first phonetics rule, the combination information and pronunciation information are determined. The syllable information is then decomposed and processed using the combination information and pronunciation information to obtain the phoneme arrangement information. Based on the analysis of phoneme arrangement information, combination information, and pronunciation information, semantic information of the educational scenario is obtained.

[0011] Preferably, the vocabulary converter performs vocabulary matching on word class information of multiple educational scenarios according to language type to obtain the converted vocabulary, specifically including the following steps: Read the second vocabulary corresponding to the language type from the vocabulary converter; The word class information of multiple educational scenarios is analyzed using natural language processing algorithms to obtain the word class meanings of multiple educational scenarios; The vocabulary converter combines the meaning of each part of speech with the second vocabulary list for analysis and judgment. If the second vocabulary contains words with the same part of speech as those used in the educational context, output the converted vocabulary. If there are no words in the second vocabulary list that have the same part of speech meaning as those in the educational context, select words with similar part of speech meanings as the conversion vocabulary.

[0012] Preferably, the step of adjusting the phonemes of the converted words using semantic information from the educational scenario to obtain the speech-to-text vocabulary for the educational scenario specifically includes the following steps: The second vocabulary is analyzed and processed to obtain second linguistic rules; The transformed vocabulary is decomposed using a second vocabulary list and second linguistic rules to obtain transformed syllable information; Based on the rules of second language, the conversion combination information and conversion pronunciation information are determined. The conversion syllable information is then decomposed and processed using the conversion combination information and conversion pronunciation information to obtain the conversion phoneme arrangement information. The phonemes of the converted words are adjusted based on semantic information, conversion combination information, and conversion pronunciation information to obtain speech-converted vocabulary for educational scenarios.

[0013] Preferably, the step of sorting the vocabulary for speech-to-text conversion in the educational scenario based on the temporal information of the educational scenario to obtain correctly converted text data specifically includes the following steps: Obtain the grammatical information corresponding to the second vocabulary; By sorting the vocabulary for speech conversion in educational scenarios using grammatical information and tense information from the educational context, the correct speech conversion sentences in educational scenarios can be obtained. The correct speech in the educational scenario is obtained by converting the speech into speech using a language converter.

[0014] Preferably, the step of sorting the vocabulary for speech conversion in the educational scenario using grammatical information and tense information of the educational scenario to obtain the correctly converted sentences in the educational scenario specifically includes the following steps: The grammatical information is parsed and processed to obtain syntactic and word order information; The vocabulary for speech-to-speech conversion in educational scenarios is sorted based on syntactic and word order information to obtain the original sentences. The original sentences are tense-adjusted based on the tense information of the educational context to obtain the correct converted sentences for the educational context.

[0015] Furthermore, a speech recognition system for educational scenarios based on subject-specific attention mechanisms is proposed to implement the aforementioned speech recognition method for educational scenarios based on subject-specific attention mechanisms, including: The speech preprocessing module is used to preprocess the speech data of the educational scenario to be recognized, and obtain the preprocessed speech data of the educational scenario. The speech segmentation module is used to segment the preprocessed educational scenario speech data to obtain multi-group segmented educational scenario speech data; The analysis module is used to analyze multi-group segmented educational scenario speech data to obtain part-of-speech information, temporal information, and semantic information of multiple educational scenarios. A language converter is used to convert speech data of an educational scenario to be identified into the language used by the learner. A vocabulary converter, which is used to convert word class information from multiple educational scenarios into converted vocabulary; The control module is used to realize information interaction between the various modules; The speech preprocessing module integrates the following: A sampling unit is used to sample the educational scene speech data to be recognized and obtain the sampled educational scene speech signal; A denoising unit is used to denoise the sampled educational scene speech signal, remove noise from the sampled educational scene speech signal, and obtain a clean speech signal; The detection unit is used to perform endpoint detection on the clean speech signal, remove the silent part of the clean speech signal, and obtain the speech signal of the target educational scenario; A normalization unit is used to normalize the target educational scene speech signal to avoid distortion of the target educational scene speech signal during the enhancement process and to obtain a normalized target educational scene speech signal. An acoustic model is used to enhance the normalized target educational scene speech signal, thereby improving the quality and identifiability of the preprocessed educational scene speech data. The speech segmentation module integrates the following: A window partitioning unit, wherein the window partitioning module is used to partition the preprocessed educational scene audio data into windows and obtain multiple sets of educational scene audio windows; The short-time energy calculation unit performs a sum of squares calculation on each educational scenario voice window to obtain multiple sets of short-time energy. A voice signal zero-crossing rate calculation unit, which calculates the zero-crossing rate of the voice signal in each educational scenario voice window to obtain the zero-crossing rate of the voice signal in the educational scenario. The judgment unit is used to judge multiple sets of short-time energy, zero-crossing rate of educational scene speech signal, and set a third threshold and a fourth threshold to determine the segmentation position of educational scene speech; The analysis module integrates the following: The extraction unit performs feature extraction on multi-group segmented educational scenario speech data based on part-of-speech information to obtain part-of-speech information for multiple educational scenarios. Construction unit, the construction unit being used to construct a first vocabulary; A rule acquisition unit is used to analyze a first vocabulary and determine first linguistic rules; The analysis unit performs temporal analysis on multi-group segmented educational scenario speech data based on word class information of multiple educational scenarios as features to obtain temporal information of the educational scenarios; the analysis unit decomposes the word class information of each educational scenario according to a first vocabulary and a first linguistic rule to obtain semantic information of the educational scenarios.

[0016] Compared with existing technologies, this invention provides a speech recognition method for educational scenarios based on subject-specific attention mechanisms, which has the following beneficial effects: This invention constructs a speech preprocessing model to preprocess the speech data of the educational scene to be identified, obtaining enhanced speech data while removing noise and silence components. A joint algorithm of short-time energy and zero-crossing rate is used to segment the preprocessed speech data into multiple complete sentences. Each sentence is analyzed to obtain phoneme arrangement information, which is then used to further extract semantic information of the educational scene. The speech data is then converted into the language used by the learner. The vocabulary of the converted speech is ordered according to the learner's grammar, and the phoneme arrangement information in the converted vocabulary is adjusted based on the semantic information of the educational scene to incorporate the teacher's emotions. Finally, a language converter converts the sentences into speech, ensuring that the speech in the teaching video contains emotion, enabling the learner to better understand the emotions contained in the language and the author's feelings when creating the text. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the speech recognition method for educational scenarios based on subject attention mechanism proposed in this invention. Figure 2 This is a structural block diagram of the speech recognition system for educational scenarios based on subject attention mechanism proposed in this invention. Detailed Implementation

[0018] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0019] Reference Figure 1As shown, the speech recognition method for educational scenarios based on subject attention mechanisms includes: Acquire the voice data of the educational scenario to be identified; It is understandable that the speech of the instructor can be collected through a dual-microphone array (sampling rate 48kHz, signal-to-noise ratio ≥60dB), while the speech data of the educational scene to be identified is the speech in the educational video. A speech preprocessing model is constructed to preprocess the speech data of the educational scenario to be recognized, thereby obtaining the preprocessed speech data of the educational scenario. Based on a joint algorithm of short-time energy and zero-crossing rate, the pre-processed educational scenario speech data is segmented to obtain multi-group segmented educational scenario speech data. Feature extraction processing is performed on multi-group segmented speech data of educational scenarios to obtain word class information and temporal information of multiple educational scenarios; The word class information of multiple educational scenarios is analyzed and processed to obtain the semantic information of the educational scenarios; Select the language type using the language converter; The vocabulary converter performs word class matching on multiple sets of educational scenarios based on language type to obtain the converted vocabulary; By adjusting the phonemes of the converted words based on the semantic information of the educational scenario, we can obtain the speech-to-vocabulary for the educational scenario. The vocabulary for speech conversion in the educational context is sorted based on the tense information of the educational context to obtain the correct speech conversion sentences in the educational context. Those skilled in the art will understand that when the instructor records educational videos in Mandarin or a dialect, and the listener can only understand the dialect or Mandarin, it is necessary to convert the audio in the educational video so that the listener can understand the meaning of the video. Therefore, the audio data in the educational video is analyzed and converted, and the converted audio is used to re-dub the educational video so that the listener can understand the emotions contained in the teacher's explanation of the text. Therefore, the audio data of the segmented educational scene is analyzed to determine the tense, word class information, and semantic information in the segmented audio data. The semantic information contains the emotions contained in the instructor's explanation of the text. The language type is selected through the audio converter, and the first vocabulary is selected according to the language type. The word class information is matched according to the first vocabulary, and words with the same or similar word class information are selected. The converted words are sorted according to the grammar of the language used by the listener. At the same time, the phoneme order in the converted words is adjusted according to the semantic information so that the converted audio contains the emotions of the instructor when explaining the text, so that the listener can understand the author's mood when creating the text. In a specific embodiment, once the conversion vocabulary is determined, weight calculations can be performed on the conversion vocabulary to determine which course the educational video is about (i.e., subject sensitivity). For example, if 90% of the conversion vocabulary is related to classical Chinese poetry, it indicates that the educational video is about the Chinese language subject. If 90% of the conversion vocabulary is mathematical formulas, then the educational video is about the mathematics subject. Therefore, the above method can also be used to determine which subject the teaching video is about, thus achieving subject recognition.

[0020] Example 1 The speech preprocessing model is used to preprocess the speech data of the educational scenario to be recognized. The specific steps to obtain the preprocessed speech data of the educational scenario are as follows: The speech data of the educational scenario to be identified is sampled and processed to obtain the sampled speech signal of the educational scenario; Acquire the frequency of the audio signal in the sampled educational scenario; Sampling processing involves adjusting the sampling rate of the educational scene speech data to be recognized to a fixed standard value, such as 8kHz, 16kHz, or 44.1kHz, to make subsequent processing more convenient. Denoising the sampled educational speech signals; If the frequency of the sampled educational scene speech signal is greater than or equal to the set first threshold, and the frequency of the sampled educational scene speech signal is less than or equal to the set second threshold, then the sampled educational scene speech signal is set as a clean speech signal. If the frequency of the sampled educational scene speech signal is less than the first threshold or greater than the second threshold, the sampled educational scene speech signal segment is removed. Endpoint detection is performed on the clean sampled educational scene speech signal to remove non-speech components and obtain the target educational scene speech signal. The non-speech components are specifically the silent part of the clean speech signal. Endpoint detection is the process of detecting the start and end points of a clean speech signal and removing non-speech components (such as silence) to reduce the complexity and computational cost of subsequent processing. Normalize the speech signal of the target educational scenario to obtain the normalized speech signal of the target educational scenario; Normalization is a process of adjusting the amplitude range of the speech signal in the target educational scenario to ensure it is within an appropriate range, thereby avoiding distortion or information loss in subsequent processing. The normalized target educational scene speech signal is enhanced by using an acoustic model to obtain preprocessed educational scene speech data; Understandably, the speech data in the educational scenarios to be identified contains linguistic components, noise components, and silence components. Therefore, it is necessary to enhance the linguistic components to make it easier for the instructor to identify the emotions contained in the text, while noise and silence components need to be removed to make subsequent processing more convenient and faster.

[0021] Example 2 Based on a joint algorithm of short-time energy and zero-crossing rate, the preprocessed educational scenario speech data is segmented to obtain multi-segmented educational scenario speech data. The specific steps include: The preprocessed educational scenario audio data is divided into windows to obtain multiple sets of educational scenario audio windows; The speech signal in each educational scenario's speech window is processed to obtain multiple sets of short-time energy, namely:

[0022] In the formula, E is the short-time energy; Let N be the value of the audio signal in the educational scenario at the nth sampling point; N is the length of the audio window in the educational scenario; and n is the number of samplings. When different sentences appear in a speech, the speech will pause. The energy at the pause is less than the energy when the speech is produced. Therefore, the energy activity is used as one of the judgment criteria. The speech signal in each educational scenario speech window is processed to obtain the zero-crossing rate of the educational scenario speech signal, i.e.:

[0023] In the formula, Z is the zero-crossing rate of the voice signal in the educational scenario; N is the length of the voice window in the educational scenario; Let n be the value of the speech signal in the educational scenario at the nth sampling point; Let be the value of the speech signal in the educational scenario at the (n-1)th sampling point; n is the number of samplings; sgn() is the sign function; The sgn() function is a sign function and has only three possible values: -1, 0, and 1. Therefore, the sgn() function is chosen to determine the zero-crossing rate of speech signals in educational scenarios. The zero-crossing rate represents the number of times a signal crosses the zero level, and is related to pauses in speech and the continuity of sound. When there is a pause in the speech The value will change, that is and If the numerical values ​​are different, and there is no pause in the speech, and The values ​​are the same. The value will not change, that is, the zero-crossing rate remains unchanged; The zero-crossing rate of multiple sets of short-time energy and educational scenario speech signals is judged and processed. If the short-term energy is greater than the set third threshold and the zero-crossing rate of the educational scene speech signal is less than the set fourth threshold, obtain the speech segmentation position in the educational scene. If the short-time energy is less than or equal to the set third threshold and the zero-crossing rate of the educational scene voice signal is greater than or equal to the set fourth threshold, output the continuous educational scene voice signal. The preprocessed educational scenario speech data is segmented by the segmentation location of the educational scenario speech to obtain multi-group segmented educational scenario speech data; Understandably, dividing the preprocessed educational scenario speech data into multiple educational scenario speech windows and calculating the short-time energy and zero-crossing rate of the educational scenario speech signal for each window shortens the data length and reduces the computational load. By judging the multiple sets of short-time energy and zero-crossing rate of the educational scenario speech signal, the segmentation position of the educational scenario speech can be determined. When the instructor pauses, it is the segmentation position of the educational scenario speech. The energy is inactive in the pause area, and the zero-crossing rate will also change. Therefore, the segmentation position of the educational scenario speech can be determined.

[0024] Example 3 The process of extracting features from multi-segmented educational scenario speech data to obtain part-of-speech and temporal information for multiple educational scenarios includes the following steps: Obtain part-of-speech information; Part-of-speech information includes criteria for noun identification, verb identification, subject identification, and adjective identification. Multiple sets of word class information are determined by extracting multi-group segmented educational scenario speech data through part-of-speech information. Using part-of-speech information as a feature, multi-group segmented speech data of educational scenarios are extracted and processed to obtain part-of-speech information of multiple educational scenarios; Using word class information from multiple educational scenarios as features, temporal analysis is performed on segmented educational scenario speech data to obtain temporal information of the educational scenarios; In addition, the multi-segmented educational scenario voice data also includes temporal information, which can specifically reflect the emotional shifts of the instructor at certain moments, enabling the listener to understand the author's mood when creating the lesson.

[0025] Example 4 Analyzing and processing word class information from multiple sets of educational scenarios to obtain semantic information specifically includes the following steps: Establish a first vocabulary, determine the first linguistic rules based on the first vocabulary, and decompose and process the information of each word class through the first vocabulary and the first linguistic rules to obtain syllable information; Based on the first phonetics rule, the combination information and pronunciation information are determined. The syllable information is then decomposed and processed using the combination information and pronunciation information to obtain the phoneme arrangement information. Vocabulary is composed of multiple syllables, and syllables are composed of multiple phonemes. Different arrangements of phonemes will result in different emotions contained in the vocabulary. Based on the analysis of phoneme arrangement information, combination information and pronunciation information, semantic information of educational scenarios is obtained; Understandably, every language has its own unique linguistic rules. By decomposing each word class information using these rules, phoneme arrangement information can be determined. Semantic information can then be derived from this phoneme arrangement information, enabling the extraction and understanding of emotions in the educational speech data to be recognized. This makes the emotions contained in the subsequent speech conversion clearer and more explicit. For example, in Chinese, words and pronunciations are composed of multiple pinyin syllables. Pinyin consists of initials, finals, and tones. Pinyin is equivalent to a syllable, while initials, finals, and tones are equivalent to phonemes. Different arrangements of initials, finals, and tones will reflect different emotions.

[0026] Example 5 The vocabulary converter performs word class matching on multiple sets of educational scenarios based on language type to obtain the converted vocabulary. The specific steps include the following: Read the second vocabulary corresponding to the language type from the vocabulary converter; The word class information of multiple educational scenarios is analyzed using natural language processing algorithms to obtain the word class meanings of multiple educational scenarios; The vocabulary converter combines the meaning of each part of speech with the second vocabulary list for analysis and judgment. If the second vocabulary contains words with the same part of speech as those used in the educational context, output the converted vocabulary. If there are no words in the second vocabulary list that have the same word class meaning as those in the educational context, select words with similar word class meanings as the conversion vocabulary. In this embodiment, due to the different languages ​​used, words that exist in Chinese may not exist in Tibetan or Mongolian. Therefore, it is necessary to match words and replace them with words that have similar meanings when no identical words exist.

[0027] Example 6 The process of adjusting the phonemes of words to be converted using semantic information from educational scenarios, and obtaining words for speech-to-text conversion in educational scenarios, specifically includes the following steps: The second vocabulary is analyzed and processed to obtain second linguistic rules; Different languages ​​have different linguistic rules. By adjusting the phoneme positions of the converted words according to the linguistic rules, the converted words in educational scenarios can be imbued with emotion. The transformed vocabulary is decomposed using a second vocabulary list and second linguistic rules to obtain transformed syllable information; Based on the rules of second language, the conversion combination information and conversion pronunciation information are determined. The conversion syllable information is then decomposed and processed using the conversion combination information and conversion pronunciation information to obtain the conversion phoneme arrangement information. The phonemes of the converted words are adjusted based on semantic information, conversion combination information, and conversion pronunciation information to obtain speech-converted vocabulary for educational scenarios. It is understandable that different languages ​​have different linguistic rules. By adjusting the phonemes of the converted words through different linguistic rules, the words in the speech conversion in the educational scenario can contain the emotions of the teacher, so that the listener can more deeply understand the author's state of mind when creating the text.

[0028] Example 7 The process of sorting and processing vocabulary for speech-to-text conversion in educational scenarios based on temporal information to obtain correctly converted text data includes the following steps: Obtain the grammatical information corresponding to the second vocabulary; By sorting the vocabulary for speech conversion in educational scenarios using grammatical information and tense information from the educational context, the correct speech conversion sentences in educational scenarios can be obtained. The correct speech in the educational scenario is obtained by converting the speech into speech using a language converter.

[0029] The process of sorting vocabulary for speech-to-text conversion in educational scenarios using grammatical and tense information to obtain correctly converted sentences includes the following steps: The grammatical information is parsed and processed to obtain syntactic and word order information; The vocabulary for speech-to-speech conversion in educational scenarios is sorted based on syntactic and word order information to obtain the original sentences. The original sentences are tense-adjusted based on the tense information of the educational context to obtain the correct converted sentences for the educational context. It is understandable that once the voice data of the educational scene to be recognized is converted, the voice data of the educational scene can be correctly converted and the voice can be re-dubbed to the educational video, because sometimes the combination of actions and voice can lead to a deeper understanding of the author's creative mindset.

[0030] Reference Figure 2 As shown, the speech recognition system for educational scenarios based on subject attention mechanisms is used to implement the speech recognition method for educational scenarios based on subject attention mechanisms described above, including: The speech preprocessing module is used to preprocess the speech data of the educational scenario to be recognized, and obtain the preprocessed speech data of the educational scenario. The speech segmentation module is used to segment the preprocessed educational scenario speech data to obtain multi-group segmented educational scenario speech data; The analysis module is used to analyze multi-group segmented educational scenario speech data to obtain part-of-speech information, temporal information, and semantic information of multiple educational scenarios. A language converter is used to convert speech data of an educational scenario to be identified into the language used by the learner. A vocabulary converter, which is used to convert word class information from multiple educational scenarios into converted vocabulary; The control module is used to realize information interaction between the various modules; The speech preprocessing module integrates the following: A sampling unit is used to sample the educational scene speech data to be recognized and obtain the sampled educational scene speech signal; A denoising unit is used to denoise the sampled educational scene speech signal, remove noise from the sampled educational scene speech signal, and obtain a clean speech signal; The detection unit is used to perform endpoint detection on the clean speech signal, remove the silent part of the clean speech signal, and obtain the speech signal of the target educational scenario; A normalization unit is used to normalize the target educational scene speech signal to avoid distortion of the target educational scene speech signal during the enhancement process and to obtain a normalized target educational scene speech signal. An acoustic model is used to enhance the normalized target educational scene speech signal, thereby improving the quality and identifiability of the preprocessed educational scene speech data. The speech segmentation module integrates the following: A window partitioning unit, wherein the window partitioning module is used to partition the preprocessed educational scene audio data into windows and obtain multiple sets of educational scene audio windows; The short-time energy calculation unit performs a sum of squares calculation on each educational scenario voice window to obtain multiple sets of short-time energy. A voice signal zero-crossing rate calculation unit, which calculates the zero-crossing rate of the voice signal in each educational scenario voice window to obtain the zero-crossing rate of the voice signal in the educational scenario. The judgment unit is used to judge multiple sets of short-time energy, zero-crossing rate of educational scene speech signal, and set a third threshold and a fourth threshold to determine the segmentation position of educational scene speech; The analysis module integrates the following: The extraction unit performs feature extraction on multi-group segmented educational scenario speech data based on part-of-speech information to obtain part-of-speech information for multiple educational scenarios. Construction unit, the construction unit being used to construct a first vocabulary; A rule acquisition unit is used to analyze a first vocabulary and determine first linguistic rules; The analysis unit performs temporal analysis on multi-group segmented educational scenario speech data based on word class information of multiple educational scenarios as features to obtain temporal information of the educational scenarios; the analysis unit decomposes the word class information of each educational scenario according to a first vocabulary and a first linguistic rule to obtain semantic information of the educational scenarios. In addition, a multimodal input module and a multimodal fusion module are also included. The multimodal input module is used to capture educational videos, while the multimodal fusion module is used to stitch features together to ensure that the speech in the educational scene is correctly converted and perfectly matched to the educational video. The multimodal input module includes an audio acquisition unit, a visual input unit, and a lesson plan parsing unit. The audio acquisition unit uses a dual-microphone array (sampling rate 48kHz, signal-to-noise ratio ≥60dB) to collect classroom speech signals. The visual input unit uses a whiteboard region detection model based on YOLOv11 to perform OCR recognition on video frames. The lesson plan parsing unit is used to parse the lesson plan PDF uploaded by the teacher (supporting LaTeX formula extraction) and perform OCR recognition on the images in the file to correct the subtitles in the educational video.

[0031] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A speech recognition method for educational scenarios based on subject-specific attention mechanisms, characterized in that, include: Acquire the voice data of the educational scenario to be identified; A speech preprocessing model is constructed to preprocess the speech data of the educational scenario to be recognized, thereby obtaining the preprocessed speech data of the educational scenario. Based on a joint algorithm of short-time energy and zero-crossing rate, the pre-processed educational scenario speech data is segmented to obtain multi-group segmented educational scenario speech data. Feature extraction processing is performed on multi-group segmented speech data of educational scenarios to obtain word class information and temporal information of multiple educational scenarios; The word class information of multiple educational scenarios is analyzed and processed to obtain the semantic information of the educational scenarios; Select the language type using the language converter; The vocabulary converter performs word class matching on multiple sets of educational scenarios based on language type to obtain the converted vocabulary; By adjusting the phonemes of the converted words based on the semantic information of the educational scenario, we can obtain the speech-to-vocabulary for the educational scenario. The vocabulary for speech conversion in educational scenarios is sorted based on the tense information of the educational scenario to obtain the correct speech conversion sentences in the educational scenario.

2. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 1, characterized in that, The process of preprocessing the educational scenario speech data to be recognized using a speech preprocessing model to obtain the preprocessed educational scenario speech data specifically includes the following steps: The speech data of the educational scenario to be identified is sampled and processed to obtain the sampled speech signal of the educational scenario; Acquire the frequency of the audio signal in the sampled educational scenario; Denoising the sampled educational speech signals; If the frequency of the sampled educational scene speech signal is greater than or equal to the set first threshold, and the frequency of the sampled educational scene speech signal is less than or equal to the set second threshold, then the sampled educational scene speech signal is set as a clean speech signal. If the frequency of the sampled educational scene speech signal is less than the first threshold or greater than the second threshold, the sampled educational scene speech signal segment is removed. Endpoint detection is performed on the clean sampled educational scene speech signal to remove non-speech components and obtain the target educational scene speech signal. The non-speech components are specifically the silent part of the clean speech signal. Normalize the speech signal of the target educational scenario to obtain the normalized speech signal of the target educational scenario; The normalized target educational scene speech signal is enhanced by using an acoustic model to obtain preprocessed educational scene speech data.

3. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 2, characterized in that, The joint algorithm based on short-time energy and zero-crossing rate is used to segment the preprocessed educational scenario speech data to obtain multi-segmented educational scenario speech data. Specifically, the steps include: The preprocessed educational scenario audio data is divided into windows to obtain multiple sets of educational scenario audio windows; The speech signal in each educational scenario's speech window is processed to obtain multiple sets of short-time energy, namely: In the formula, E is the short-time energy; Let N be the value of the audio signal in the educational scenario at the nth sampling point; N is the length of the audio window in the educational scenario; and n is the number of samplings. The speech signal in each educational scenario speech window is processed to obtain the zero-crossing rate of the educational scenario speech signal, i.e.: In the formula, Z is the zero-crossing rate of the voice signal in the educational scenario; N is the length of the voice window in the educational scenario; Let n be the value of the speech signal in the educational scenario at the nth sampling point; Let be the value of the speech signal in the educational scenario at the (n-1)th sampling point; n is the number of samplings; sgn() is the sign function; The zero-crossing rate of multiple sets of short-time energy and educational scenario speech signals is judged and processed. If the short-term energy is greater than the set third threshold and the zero-crossing rate of the educational scene speech signal is less than the set fourth threshold, obtain the speech segmentation position in the educational scene. If the short-time energy is less than or equal to the set third threshold and the zero-crossing rate of the educational scene voice signal is greater than or equal to the set fourth threshold, output the continuous educational scene voice signal. The pre-processed educational scenario speech data is segmented by the location of the speech segments in the educational scenario to obtain multi-group segmented educational scenario speech data.

4. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 3, characterized in that, The process of extracting features from multi-segmented educational scenario speech data to obtain part-of-speech and temporal information for multiple educational scenarios specifically includes the following steps: Obtain part-of-speech information; Using part-of-speech information as a feature, multi-group segmented speech data of educational scenarios are extracted and processed to obtain part-of-speech information of multiple educational scenarios; Using word class information from multiple educational scenarios as features, temporal analysis is performed on segmented educational scenario speech data to obtain temporal information of the educational scenarios.

5. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 4, characterized in that, The process of analyzing and processing word class information from multiple sets of educational scenarios to obtain semantic information specifically includes the following steps: Establish a first vocabulary, determine the first linguistic rules based on the first vocabulary, and decompose and process the information of each word class through the first vocabulary and the first linguistic rules to obtain syllable information; Based on the first phonetics rule, the combination information and pronunciation information are determined. The syllable information is then decomposed and processed using the combination information and pronunciation information to obtain the phoneme arrangement information. Based on the analysis of phoneme arrangement information, combination information, and pronunciation information, semantic information of the educational scenario is obtained.

6. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 5, characterized in that, The vocabulary converter performs word class matching on multiple sets of educational scenarios based on language type to obtain the converted vocabulary. The specific steps include the following: Read the second vocabulary corresponding to the language type from the vocabulary converter; The word class information of multiple educational scenarios is analyzed using natural language processing algorithms to obtain the word class meanings of multiple educational scenarios; The vocabulary converter combines the meaning of each part of speech with the second vocabulary list for analysis and judgment. If the second vocabulary contains words with the same part of speech as those used in the educational context, output the converted vocabulary. If there are no words in the second vocabulary list that have the same part of speech meaning as those in the educational context, select words with similar part of speech meanings as the conversion vocabulary.

7. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 6, characterized in that, The process of adjusting the phonemes of the converted vocabulary based on semantic information from the educational context to obtain the speech-to-text vocabulary for the educational context includes the following steps: The second vocabulary is analyzed and processed to obtain second linguistic rules; The transformed vocabulary is decomposed using a second vocabulary list and second linguistic rules to obtain transformed syllable information; Based on the rules of second language, the conversion combination information and conversion pronunciation information are determined. The conversion syllable information is then decomposed and processed using the conversion combination information and conversion pronunciation information to obtain the conversion phoneme arrangement information. The phonemes of the converted words are adjusted based on semantic information, conversion combination information, and conversion pronunciation information to obtain speech-converted vocabulary for educational scenarios.

8. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 7, characterized in that, The process of sorting the vocabulary for speech-to-text conversion in the educational context based on the temporal information of the educational scenario to obtain correctly converted text data specifically includes the following steps: Obtain the grammatical information corresponding to the second vocabulary; By sorting the vocabulary for speech conversion in educational scenarios using grammatical information and tense information from the educational context, the correct speech conversion sentences in educational scenarios can be obtained. The correct speech in the educational scenario is obtained by converting the speech into speech using a language converter.

9. The speech recognition method for educational scenarios based on subject attention mechanism according to claim 8, characterized in that, The process of sorting vocabulary for speech-to-text conversion in educational scenarios using grammatical information and tense information of the educational context to obtain correctly converted sentences specifically includes the following steps: The grammatical information is parsed and processed to obtain syntactic and word order information; The vocabulary for speech-to-speech conversion in educational scenarios is sorted based on syntactic and word order information to obtain the original sentences. The original sentences are tense-adjusted based on the tense information of the educational context to obtain the correct converted sentences for the educational context.

10. A speech recognition system for educational scenarios based on subject-specific attention mechanisms, used to implement the speech recognition method for educational scenarios based on subject-specific attention mechanisms as described in any one of claims 1-9, characterized in that, include: The speech preprocessing module is used to preprocess the speech data of the educational scenario to be recognized, and obtain the preprocessed speech data of the educational scenario. The speech segmentation module is used to segment the preprocessed educational scenario speech data to obtain multi-group segmented educational scenario speech data; The analysis module is used to analyze multi-group segmented educational scenario speech data to obtain part-of-speech information, temporal information, and semantic information of multiple educational scenarios. A language converter is used to convert speech data of an educational scenario to be identified into the language used by the learner. A vocabulary converter, which is used to convert word class information from multiple educational scenarios into converted vocabulary; The control module is used to realize information interaction and data transmission between various modules; The speech preprocessing module integrates the following: A sampling unit is used to sample the educational scene speech data to be recognized and obtain the sampled educational scene speech signal; A denoising unit is used to denoise the sampled educational scene speech signal, remove noise from the sampled educational scene speech signal, and obtain a clean speech signal; The detection unit is used to perform endpoint detection on the clean speech signal, remove the silent part of the clean speech signal, and obtain the speech signal of the target educational scenario; A normalization unit is used to normalize the target educational scene speech signal to avoid distortion of the target educational scene speech signal during the enhancement process and to obtain a normalized target educational scene speech signal. An acoustic model is used to enhance the normalized target educational scene speech signal, thereby improving the quality and identifiability of the preprocessed educational scene speech data. The speech segmentation module integrates the following: A window partitioning unit, wherein the window partitioning module is used to partition the preprocessed educational scene audio data into windows and obtain multiple sets of educational scene audio windows; The short-time energy calculation unit performs a sum of squares calculation on each educational scenario voice window to obtain multiple sets of short-time energy. A voice signal zero-crossing rate calculation unit, which calculates the zero-crossing rate of the voice signal in each educational scenario voice window to obtain the zero-crossing rate of the voice signal in the educational scenario. The judgment unit is used to judge multiple sets of short-time energy, zero-crossing rate of educational scene speech signal, and set a third threshold and a fourth threshold to determine the segmentation position of educational scene speech; The analysis module integrates the following: The extraction unit performs feature extraction on multi-group segmented educational scenario speech data based on part-of-speech information to obtain part-of-speech information for multiple educational scenarios. Construction unit, the construction unit being used to construct a first vocabulary; A rule acquisition unit is used to analyze a first vocabulary and determine first linguistic rules; The analysis unit performs temporal analysis on multi-group segmented educational scenario speech data based on word class information of multiple educational scenarios as features to obtain temporal information of the educational scenarios; the analysis unit decomposes the word class information of each educational scenario according to a first vocabulary and a first linguistic rule to obtain semantic information of the educational scenarios.

Images