Interdisciplinary term library adaptive speech recognition method, device and equipment
By integrating the features of subject-specific hot words database and PPT content in a classroom environment, and using pre-trained models and deep learning technology for speech recognition, the system solves the problem of misjudgment in recognizing specific subject-specific vocabulary and homophones in traditional systems, thereby improving the accuracy and robustness of speech recognition in teaching scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional speech recognition systems struggle to understand subject-specific vocabulary and handle cross-disciplinary homonyms in classroom settings, resulting in high misjudgment rates, especially in noisy environments and multi-person conversations where recognition accuracy is poor.
By acquiring subject-specific hot word databases and structured PPT content, pre-trained word segmentation and word embedding models are used to extract text features. Subject-specific labels are determined by combining adversarial autoencoders and multi-head self-attention mechanisms. Subject-specific terminology databases are dynamically loaded for speech recognition, and features from real-time speech context and structured PPT content are integrated.
It improves the accuracy of speech recognition, especially in the understanding and recognition of subject-specific vocabulary in teaching scenarios, adapts to rapid switching between multi-disciplinary scenarios, and enhances the robustness and professionalism of speech recognition.
Smart Images
Figure CN121662046A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of speech recognition, and more specifically, to a method, apparatus, device, and storage medium for adaptive speech recognition based on an interdisciplinary terminology database. Background Technology
[0002] The application of speech recognition technology in classroom settings has profound significance, as the generated text transcripts greatly facilitate student review and teacher recollection. However, the unique environment of offline classrooms (background noise, multi-person dialogue, far-field recording) combined with the inherent complexity of teaching content, especially the highly specialized and significantly different terminology systems of different disciplines, poses a severe challenge to traditional speech recognition systems in such scenarios. Systems often lack understanding of the specific vocabulary of particular disciplines (such as legal terminology, biological Latin names, programming language keywords, etc.) or are subject to interference from homophones across disciplines (such as "torque" and "example sentence"), easily leading to misjudgments of unfamiliar or homophone words by the recognition model, inevitably increasing the error rate. Summary of the Invention
[0003] To overcome at least one of the defects described in the prior art, this invention provides an adaptive speech recognition method, apparatus, device, and storage medium with an interdisciplinary terminology database. The technical solution adopted by this invention is as follows.
[0004] In a first aspect, the present invention provides an adaptive speech recognition method based on an interdisciplinary terminology database, comprising:
[0005] Obtain a subject-specific hot term database, which includes: terms and subject tags corresponding to each term;
[0006] Retrieve the recognized text from n seconds prior to the current time, and retrieve the title, bolded text, and full-page text of the PPT presentation at the current time.
[0007] Based on the recognized text, text features are extracted through the text content processing module to generate the preceding speech text feature vector.
[0008] Based on the titles, bold text, and full-page text of the PPT content, text features are extracted through the text content processing module to obtain the PPT structured text feature vector;
[0009] Based on the aforementioned speech text feature vector and PPT structured text feature vector, output subject labels;
[0010] Based on the subject tags, obtain the corresponding terms from the subject hot word library, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio;
[0011] The process of extracting text features through the text content processing module includes:
[0012] The input text is segmented using a pre-trained word segmentation model. Meaningless words in the stop word list are removed, and meaningful words are retained and recorded as the remaining terms.
[0013] Based on the remaining terms, the weights of each remaining term are calculated using the TF-IDF values obtained from the statistics of a large-scale pre-training corpus and the word frequencies in the current text, which are dynamically calculated.
[0014] Use a pre-trained word embedding model to convert each remaining term into a word vector;
[0015] When extracting text features from the identified text, a text feature vector is generated based on the word vectors and according to the weights of each remaining term.
[0016] When extracting text features from the title, bold text, and full-page text of the PPT content, a text feature vector is generated based on the word vectors, according to the weights of each remaining term, and by adding the weights of the title and bold text of the PPT content.
[0017] In one implementation, the process of calculating the weight of each retained term based on the retained terms, according to the TF-IDF values obtained from large-scale pre-training corpus statistics and the dynamically calculated word frequencies in the current text, includes:
[0018] The remaining terms are used to obtain the TF-IDF value table v1 from large-scale corpus statistics, the importance score weights of manually annotated terms are obtained v2, and the word frequency of the remaining terms in the current text is dynamically calculated v3. These three are combined to obtain the final weight w of the term.
[0019] w = v3 * (a * v2 + (1 - a) * v1);
[0020] Where 'a' represents the preset subject preference weight.
[0021] In one implementation, prior to the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the method further includes:
[0022] For the remaining terms whose weights are higher than the preset term weight threshold, word vector similarity is calculated based on the pre-trained word embedding model, and then K-Means clustering is used to select the center word of the largest cluster as the core keyword group.
[0023] In the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the core keyword groups are converted into word vectors using the pre-trained word embedding model.
[0024] In one implementation, the process of outputting subject labels based on the preceding speech text feature vector and the PPT structured text feature vector includes:
[0025] The aforementioned speech text feature vector and PPT structured text feature vector are input into the subject orientation prediction model to determine the subject correlation and obtain the probability distribution of belonging to each subject.
[0026] When the highest probability value of a subject is greater than the probability threshold, and the difference between the highest probability value and the second highest probability value is greater than the difference threshold, the subject corresponding to the highest probability value is output as the subject label. Here, the first probability threshold is greater than the second probability threshold.
[0027] The subject-preference prediction model includes:
[0028] The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, which are both of the first dimension. The feature fusion and compression are performed by an adversarial autoencoder to output a fused latent feature vector of the second dimension, where the second dimension is smaller than the first dimension.
[0029] A deep semantic extraction layer is used to perform deep semantic modeling on the latent feature vectors using a multi-head self-attention mechanism combined with residual connections.
[0030] The result prediction layer is used to calculate the probability distribution of each subject using a fully connected layer and a Softmax function.
[0031] In one implementation, the process of obtaining the corresponding terms from the subject hot word library based on the subject tag, adding the obtained terms to the speech recognition decoding process, and outputting the speech recognition result of the current input audio includes:
[0032] Based on the subject tags, retrieve the terminology list of the corresponding subject from the subject hot word database;
[0033] Each term in the term list is converted into a word vector using a pre-trained word embedding model;
[0034] Perform average pooling on all word vectors to generate a subject-specific hot word aggregation feature vector H;
[0035] The speech recognition acoustic model processes the current audio frame and outputs a frame-level acoustic feature sequence A.
[0036] The frame-level acoustic feature sequence A and the subject hot word aggregation feature vector H are passed through a multi-head cross-attention module to generate an enhanced acoustic feature E that incorporates hot word information;
[0037] The enhanced acoustic feature E is input into the speech recognition decoder, and the final recognized text is output.
[0038] In one implementation, it further includes:
[0039] Obtain a terminology mapping table related to a predefined discipline, wherein the Key in the terminology mapping table is a colloquial expression and the Value is a standardized written expression;
[0040] The speech recognition results are corrected based on the terminology mapping table.
[0041] In one implementation, the preceding speech text feature vector is 128-dimensional, and the PPT structured text feature vector is 128-dimensional.
[0042] The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, resulting in a 256-dimensional concatenation. The feature fusion and compression are performed through an adversarial autoencoder to output a fused 64-dimensional latent feature vector.
[0043] Secondly, the present invention provides an adaptive speech recognition device with an interdisciplinary terminology database, comprising:
[0044] The acquisition module is used to acquire a subject-specific hot term database, which includes: terms and their corresponding subject tags; acquire the recognized text from n seconds prior to the current time point; and acquire the title, bolded text, and full-page text of the PPT content at the current time point.
[0045] The extraction module is used to extract text features based on the recognized text through the text content processing module, and generate the preceding speech text feature vector; based on the title, bold text and full-page text of the PPT content, the text content processing module extracts text features to obtain the PPT structured text feature vector.
[0046] The determination module is used to output subject labels based on the feature vectors of the preceding speech text and the feature vectors of the PPT structured text;
[0047] The recognition module is used to obtain the corresponding terms in the subject hot word library according to the subject tag, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio.
[0048] The extraction module performs the process of extracting text features through the text content processing module, including:
[0049] The input text is segmented using a pre-trained word segmentation model. Meaningless words in the stop word list are removed, and meaningful words are retained and recorded as the remaining terms.
[0050] Based on the remaining terms, the weights of each remaining term are calculated using the TF-IDF values obtained from the statistics of a large-scale pre-training corpus and the word frequencies in the current text, which are dynamically calculated.
[0051] Use a pre-trained word embedding model to convert each remaining term into a word vector;
[0052] When extracting text features from the identified text, a text feature vector is generated based on the word vectors and according to the weights of each remaining term.
[0053] When extracting text features from the title, bold text, and full-page text of the PPT content, a text feature vector is generated based on the word vectors, according to the weights of each remaining term, and by adding the weights of the title and bold text of the PPT content.
[0054] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.
[0055] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method of any of the above embodiments.
[0056] This invention utilizes multimodal context perception, which integrates the dual-channel features of real-time speech context and structured PPT content, to drive subject-specific context determination. Then, it performs targeted speech recognition based on a corresponding subject-specific terminology database. This invention improves recognition accuracy and enhances speech recognition performance in teaching scenarios by understanding the specific vocabulary of a particular subject. Attached Figure Description
[0057] Figure 1 This is a schematic diagram of the overall process of one embodiment of the present invention.
[0058] Figure 2 This is a schematic diagram of the process for extracting text features according to Embodiment 1 of the present invention.
[0059] Figure 3 This is a schematic diagram of the overall structure of Embodiment 2 of the present invention. Detailed Implementation
[0060] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0061] It should be noted that the terms "first, second, ..." used in the embodiments of the present invention are merely used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, ..." can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first, second, ..." can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0062] Example 1
[0063] Please see Figure 1 , Figure 1 This is a flowchart illustrating an adaptive speech recognition method based on an interdisciplinary terminology database according to Embodiment 1 of the present invention. The method includes steps S110, S120, S130, S140, S150, and S160. It should be noted that steps S110, S120, S130, S140, S150, and S160 are merely reference numerals used to clearly explain the embodiment and the accompanying drawings. Figure 1 The correspondence is not intended to limit the order of steps in this embodiment.
[0064] Step S110: Obtain a subject hot word database, wherein the subject hot word database includes: terms and subject tags corresponding to each term;
[0065] Step S120: Obtain the recognized text from n seconds prior to the current time point, and obtain the title, bolded text, and full-page text of the PPT content at the current time point.
[0066] Step S130: Based on the recognized text, extract text features through the text content processing module to generate the preceding speech text feature vector;
[0067] Step S140: Based on the title, bold text, and full-page text of the PPT content, extract text features through the text content processing module to obtain the PPT structured text feature vector;
[0068] Step S150: Output subject labels based on the aforementioned speech text feature vector and PPT structured text feature vector;
[0069] Step S160: Obtain the corresponding terms from the subject hot word library according to the subject tag, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio.
[0070] Among them, such as Figure 2 As shown, Figure 2This is a flowchart illustrating the process of extracting text features according to Embodiment 1 of the present invention. The process of extracting text features through the text content processing module includes: steps S210, S220, S230, S240, and S250.
[0071] Step S210: Use a pre-trained word segmentation model to segment the input text, remove meaningless words from the stop word list, and retain meaningful words, which are recorded as the remaining terms.
[0072] Step S220: Based on the remaining terms, calculate the weight of each remaining term according to the TF-IDF value obtained from the statistics of the large-scale pre-training corpus and the word frequency in the current text obtained by dynamic calculation.
[0073] Step S230: Use a pre-trained word embedding model to convert each remaining term into a word vector;
[0074] Step S240: When extracting text features of the identified text, a text feature vector is generated based on the word vector and according to the weight of each remaining term.
[0075] Step S250: When extracting text features of the title, bold text, and full-page text of the PPT content, based on the word vectors, according to the weights of each remaining term, and by increasing the weights of the title and bold text of the PPT content, a text feature vector is generated.
[0076] Before step S110, specialized terms and frequently used words from different disciplines are collected to construct a subject-specific hot word database containing the terms and their respective subject tags. Step S110 involves acquiring this subject-specific hot word database. Establishing a multidisciplinary knowledge base that quantifies the importance of terms can provide data support for subsequent context-driven dynamic enhancement of terms.
[0077] This method breaks away from the traditional reliance on a single audio signal. It integrates temporal context (i.e., speech data) and spatial context (i.e., the teacher's PPT data) to determine the subject being taught. Then, based on the determined subject, it retrieves the corresponding subject vocabulary, thereby improving the accuracy of speech recognition.
[0078] Step S120 involves acquiring sample data for determining the subject. Regarding the audio data, audio-based text extraction utilizes a speech recognition module to acquire and cache the recognized text from n seconds prior to the current time (e.g., n = 3-10). Regarding the teacher's PPT data, structured PPT content extraction uses a PPT content parsing module to extract the title, bolded text, and full-page text of the PPT content at the current time.
[0079] Steps S130 and S140 respectively send the extracted recognized text, as well as the title, bold text, and full-page text of the PPT content, into the text content processing module for processing to obtain the preceding speech text feature vector T1 (128-dimensional vector) about the recognized text and the PPT structured text feature vector T2 (128-dimensional vector) about the PPT content.
[0080] The specific text content processing module's process for extracting text features is as follows: Figure 2 As shown.
[0081] Step S210 is word segmentation and cleaning. A pre-trained word segmentation model (such as Jieba or BPE) is used to segment the input text, remove meaningless words from the stop word list, and retain meaningful words, which are recorded as the remaining terms.
[0082] A pre-trained tokenization model is a pre-trained model specifically designed to break down natural language text into meaningful linguistic units (such as words, subwords, or characters). Unlike traditional rule-based or dictionary-based tokenization methods, it utilizes machine learning (especially deep learning) models trained on large-scale corpora, enabling it to handle text segmentation problems more intelligently and flexibly, particularly performing better in handling out-of-vocabulary words, ambiguous segmentation, and multilingual scenarios.
[0083] In step S220, the remaining terms are retrieved from a large-scale corpus to obtain the TF-IDF value table v1. Then (if available) the importance score weights of manually annotated terms v2 are obtained, and the word frequency v3 of the remaining terms in the current text is dynamically calculated. These three factors are then combined to obtain the final weight of the term.
[0084] w = v3 * (a * v2 + (1 - a) * v1);
[0085] Where 'a' represents the subject preference weight, assigning higher priority to subjects (such as core subjects like Chinese, Mathematics, and English); if the remaining term is not in the TF-IDF value table, then the average value of the TF-IDF value table is taken.
[0086] The TF-IDF (Term Frequency-Inverse Document Frequency) mentioned above is a statistical metric used to measure the importance of a word in a collection of documents. It is widely used in information retrieval, text mining, and natural language processing, and is one of the fundamental methods for text feature representation. Its core idea is: the more frequently a word appears in a document (high TF) and the less frequently it appears in all documents (high IDF), the more representative that word is of that document, and the higher its TF-IDF value.
[0087] Step S230 uses a pre-trained word embedding model to convert each remaining term into a word vector.
[0088] A pre-trained word embedding model is a word vector representation model that has been trained on large-scale text corpora and can be used directly. It maps each word to a dense vector of fixed dimensions, making words with similar semantics or grammar closer together in the vector space.
[0089] Key features:
[0090] Pre-training:
[0091] The models are trained by research institutions or companies (such as Google, Facebook, and OpenAI) on massive amounts of data (such as Wikipedia, news, and web pages), so users do not need to train them from scratch and can directly download and use them.
[0092] Low-dimensional dense vectors:
[0093] Each word is represented as a real vector of tens to hundreds of dimensions (e.g., 300 dimensions), which significantly reduces the curse of dimensionality compared to traditional one-hot encoding (high-dimensional sparse).
[0094] Semantic capture:
[0095] Co-occurrence relationships of words are learned through context, and the distance in the vector space reflects the semantic similarity of words (e.g., the vectors of "king" and "queen" are close).
[0096] Step S240 corresponds to step S130, which involves extracting text features from the identified text through the text content processing module. At this point, based on the word vectors obtained in step S230, a weighted average of the remaining terms obtained in step S220 is used to generate the final text feature vector (128 dimensions).
[0097] Step S250 corresponds to step S140, which involves extracting text features from the title, bold text, and full-page text of the PPT content through the text content processing module. At this point, based on the word vectors obtained in step S230, and using a weighted average of the remaining terms obtained in step S220, the title and bold text of the content are given even greater weights during text feature calculation to generate the final text feature vector (128 dimensions).
[0098] In one implementation, before step S230, which describes the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the method further includes step S221.
[0099] Step S221: For the remaining terms whose weights are higher than the preset term weight threshold, calculate the word vector similarity based on the pre-trained word embedding model, and then use K-Means clustering to select the center word of the largest cluster as the core keyword group.
[0100] In the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the core keyword groups are converted into word vectors using the pre-trained word embedding model.
[0101] In this embodiment, a keyword aggregation step is added: for the selected high-weight terms, word vector similarity is calculated based on a pre-trained word embedding model (such as Word2Vec, BERT), and K-Means clustering is used to select the center word of the largest cluster as the core keyword group. Then, step S230 uses the core keyword group to convert and obtain word vectors.
[0102] This allows us to focus more on key vocabulary when judging a subject.
[0103] In step S150, the preceding speech text feature vector T1 and the PPT structured text feature vector T2 are input into a pre-trained model (such as a traditional machine learning model, a deep learning model, and a pre-trained language model) to determine the subject relevance and output the current semantic relevance subject category result.
[0104] Step S150 accurately determines the core subject context of the current semantic scene based on the fused multimodal features, providing a decision basis for dynamically loading subject-specific terms.
[0105] In one implementation, step S150, which involves outputting subject labels based on the preceding speech text feature vector and the PPT structured text feature vector, includes steps S151 and S152.
[0106] Step S151: Input the aforementioned speech text feature vector and PPT structured text feature vector into the subject tendency prediction model to determine the subject correlation and obtain the probability distribution of belonging to each subject.
[0107] Step S152: When the highest subject probability value is greater than the probability threshold and the difference between it and the second highest probability value is greater than the difference threshold, the subject corresponding to the highest subject probability value is output as the subject label, wherein the first probability threshold is greater than the second probability threshold.
[0108] The subject-preference prediction model includes:
[0109] The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, which are both of the first dimension. The feature fusion and compression are performed by an adversarial autoencoder to output a fused latent feature vector of the second dimension, where the second dimension is smaller than the first dimension.
[0110] A deep semantic extraction layer is used to perform deep semantic modeling on the latent feature vectors using a multi-head self-attention mechanism combined with residual connections.
[0111] The result prediction layer is used to calculate the probability distribution of each subject using a fully connected layer and a Softmax function.
[0112] In step S151, a subject-matter preference prediction model is used to determine the subject-matter correlation. The subject-matter preference prediction model in this embodiment is a modified propensity model. A propensity model is a type of statistical or machine learning model that predicts an individual's future behavior or choice preferences based on historical data; its core objective is to estimate the probability of a certain behavior occurring.
[0113] Specifically, the subject-oriented prediction model in this implementation includes: a feature fusion and dimensionality reduction layer, a deep semantic extraction layer, and a result prediction layer.
[0114] Feature fusion and dimensionality reduction layer: T1 and T2 (256 dimensions in total) are concatenated, and feature fusion and compression are performed by an adversarial autoencoder (AAE) to output a fused low-dimensional latent feature vector (64 dimensions).
[0115] Deep semantic extraction layer: For 64-dimensional latent features, a multi-head self-attention mechanism combined with residual connections is used to perform deep semantic modeling, which enhances the model's sensitivity to local technical terminology features.
[0116] Result prediction layer: The probability distribution of each subject is calculated using a fully connected layer (FC layer) and a softmax function.
[0117] After step S151, which yields the probability distribution of each subject, step S152 outputs subject labels based on certain judgment logic.
[0118] The final output result judgment logic is as follows: when the highest subject probability value is greater than the threshold θ1 (e.g., θ1 = 0.6), and the difference between the highest and second-highest subject probability values is greater than the threshold θ2 (e.g., θ2 = 0.2), the subject corresponding to the highest subject probability value is output as the subject label. Otherwise, it indicates that it is difficult to distinguish which subject it is, and the label "unknown" is output.
[0119] After step S150, it can be determined what subject the class is in. In step S160, the corresponding professional terminology of the current subject is called from the subject hot word library during the speech recognition process, and then the speech recognition result of the audio is output to improve the accuracy of the speech recognition result.
[0120] In one embodiment, step S160, which involves obtaining the corresponding terms from the subject hot word library based on the subject tag, adding the obtained terms to the speech recognition decoding process, and outputting the speech recognition result of the current input audio, includes steps S161, S162, S163, S164, S165, and S166.
[0121] Step S161: Based on the subject tag, retrieve the terminology list of the corresponding subject from the subject hot word database;
[0122] Step S162: Convert each term in the term list into a word vector using a pre-trained word embedding model;
[0123] Step S163: Perform average pooling on all word vectors to generate subject hot word aggregation feature vector H;
[0124] Step S164: The speech recognition acoustic model processes the current audio frame and outputs a frame-level acoustic feature sequence A.
[0125] Step S165: The frame-level acoustic feature sequence A and the subject hot word aggregation feature vector H are passed through a multi-head cross-attention module to generate an enhanced acoustic feature E that integrates hot word information;
[0126] Step S166: Input the enhanced acoustic feature E into the speech recognition decoder and output the final recognized text.
[0127] This implementation method is a specific speech recognition process with hot words, which includes four major processes: hot word feature extraction, backbone acoustic feature extraction, feature fusion, and recognition decoding.
[0128] Hot word feature extraction: Based on the subject labels output in step S150, retrieve the term list of the corresponding subject from the dynamic subject hot word library; convert each term into a word vector through a pre-trained word embedding model; perform average pooling on all word vectors in the term list to generate the subject hot word aggregated feature vector H.
[0129] Main acoustic feature extraction: The speech recognition acoustic model processes the current audio frame and outputs a frame-level acoustic feature sequence A.
[0130] Feature fusion: The frame-level acoustic feature sequence A and the subject hot word aggregation feature vector H are processed through the multi-head cross-attention (MHA) module to generate an enhanced acoustic feature E that incorporates hot word information.
[0131] Recognition and Decoding: The enhanced feature sequence E is input into the speech recognition decoder (such as CTC / RNN-T) and the final recognized text is output.
[0132] Based on a general and stable speech recognition backbone model, this implementation method dynamically injects subject-specific contextual information into acoustic features through a lightweight and pluggable cross-modal attention fusion module, which significantly improves the robustness of recognition of terms in specific fields and can adapt to rapid switching in multi-disciplinary scenarios.
[0133] This method utilizes multimodal context awareness, which integrates dual-channel features of real-time speech context and structured PPT content, to drive subject-specific context determination. Then, it performs targeted speech recognition based on a corresponding subject-specific terminology database. This method improves recognition accuracy and enhances speech recognition performance in teaching scenarios by understanding the specific vocabulary of a particular subject.
[0134] In one embodiment, the interdisciplinary terminology-based adaptive speech recognition method further includes steps S170 and S180.
[0135] Step S170: Obtain a predefined subject-related terminology mapping table, wherein the Key in the terminology mapping table is a colloquial expression and the Value is a standardized written expression.
[0136] Step S180: Correct the speech recognition result according to the terminology mapping table.
[0137] In this implementation, a predefined subject-related terminology mapping table is constructed (Key: colloquial expression / Value: standardized written form).
[0138] Example:
[0139] (Chemistry) "Fe3O4" is a chemical compound that turns into "Fe3O4".
[0140] (Mathematics) "Ciensita" -> "sinθ";
[0141] (Computer) "Bool value" -> "bool".
[0142] Then, during output, more professional terminology is used to resolve the differences in professional expression between spoken and written language, thereby improving the professionalism and consistency of the output content.
[0143] Example 2
[0144] Corresponding to the method in Example 1, such as Figure 3 As shown, the present invention also provides a classroom facial recognition attendance device 3, comprising: an acquisition module 310, an extraction module 320, a judgment module 330, and a recognition module 340.
[0145] The acquisition module 310 is used to acquire a subject-specific hot word library, wherein the subject-specific hot word library includes: terms, subject tags corresponding to each term; acquire the recognized text from n seconds prior to the current time point, and acquire the title, bold text, and full-page text of the PPT content at the current time point.
[0146] The extraction module 320 is used to extract text features based on the recognized text through the text content processing module to generate the preceding speech text feature vector; based on the title, bold text and full-page text of the PPT content, it extracts text features through the text content processing module to obtain the PPT structured text feature vector.
[0147] The determination module 330 is used to output subject labels based on the preceding speech text feature vector and the PPT structured text feature vector;
[0148] The recognition module 340 is used to obtain the corresponding terms in the subject hot word library according to the subject tag, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio.
[0149] The extraction module performs the process of extracting text features through the text content processing module, including:
[0150] The input text is segmented using a pre-trained word segmentation model. Meaningless words in the stop word list are removed, and meaningful words are retained and recorded as the remaining terms.
[0151] Based on the remaining terms, the weights of each remaining term are calculated using the TF-IDF values obtained from the statistics of a large-scale pre-training corpus and the word frequencies in the current text, which are dynamically calculated.
[0152] Use a pre-trained word embedding model to convert each remaining term into a word vector;
[0153] When extracting text features from the identified text, a text feature vector is generated based on the word vectors and according to the weights of each remaining term.
[0154] When extracting text features from the title, bold text, and full-page text of the PPT content, a text feature vector is generated based on the word vectors, according to the weights of each remaining term, and by adding the weights of the title and bold text of the PPT content.
[0155] In one implementation, the process of calculating the weight of each retained term based on the retained terms, according to the TF-IDF values obtained from large-scale pre-training corpus statistics and the dynamically calculated word frequencies in the current text, includes:
[0156] The remaining terms are used to obtain the TF-IDF value table v1 from large-scale corpus statistics, the importance score weights of manually annotated terms are obtained v2, and the word frequency of the remaining terms in the current text is dynamically calculated v3. These three are combined to obtain the final weight w of the term.
[0157] w = v3 * (a * v2 + (1 - a) * v1);
[0158] Where 'a' represents the preset subject preference weight.
[0159] In one implementation, prior to the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the method further includes:
[0160] For the remaining terms whose weights are higher than the preset term weight threshold, word vector similarity is calculated based on the pre-trained word embedding model, and then K-Means clustering is used to select the center word of the largest cluster as the core keyword group.
[0161] In the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the core keyword groups are converted into word vectors using the pre-trained word embedding model.
[0162] In one implementation, the determination module performs the process of outputting subject labels based on the preceding speech text feature vector and the PPT structured text feature vector, including:
[0163] The aforementioned speech text feature vector and PPT structured text feature vector are input into the subject orientation prediction model to determine the subject correlation and obtain the probability distribution of belonging to each subject.
[0164] When the highest probability value of a subject is greater than the probability threshold, and the difference between the highest probability value and the second highest probability value is greater than the difference threshold, the subject corresponding to the highest probability value is output as the subject label. Here, the first probability threshold is greater than the second probability threshold.
[0165] The subject-preference prediction model includes:
[0166] The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, which are both of the first dimension. The feature fusion and compression are performed by an adversarial autoencoder to output a fused latent feature vector of the second dimension, where the second dimension is smaller than the first dimension.
[0167] A deep semantic extraction layer is used to perform deep semantic modeling on the latent feature vectors using a multi-head self-attention mechanism combined with residual connections.
[0168] The result prediction layer is used to calculate the probability distribution of each subject using a fully connected layer and a Softmax function.
[0169] In one implementation, the recognition module performs the process of obtaining the corresponding terms from the subject hot word library based on the subject tag, adding the obtained terms to the speech recognition decoding process, and outputting the speech recognition result of the current input audio, including:
[0170] Based on the subject tags, retrieve the terminology list of the corresponding subject from the subject hot word database;
[0171] Each term in the term list is converted into a word vector using a pre-trained word embedding model;
[0172] Perform average pooling on all word vectors to generate a subject-specific hot word aggregation feature vector H;
[0173] The speech recognition acoustic model processes the current audio frame and outputs a frame-level acoustic feature sequence A.
[0174] The frame-level acoustic feature sequence A and the subject hot word aggregation feature vector H are passed through a multi-head cross-attention module to generate an enhanced acoustic feature E that incorporates hot word information;
[0175] The enhanced acoustic feature E is input into the speech recognition decoder, and the final recognized text is output.
[0176] In one embodiment, the recognition module is further configured to obtain a predefined subject-related terminology mapping table, wherein the Key in the terminology mapping table is a colloquial expression and the Value is a standardized written expression; and to correct the speech recognition result based on the terminology mapping table.
[0177] In one implementation, the preceding speech text feature vector is 128-dimensional, and the PPT structured text feature vector is 128-dimensional.
[0178] The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, resulting in a 256-dimensional concatenation. The feature fusion and compression are performed through an adversarial autoencoder to output a fused 64-dimensional latent feature vector.
[0179] This device utilizes multimodal context perception, which integrates real-time speech context and structured PPT content into a dual-channel feature set to drive subject-specific context determination. Then, it performs targeted speech recognition based on a corresponding subject-specific terminology database. This device can improve recognition accuracy and enhance speech recognition performance in teaching scenarios by understanding the specific vocabulary of a particular subject.
[0180] Example 3
[0181] This invention also provides a storage medium storing computer instructions that, when executed by a processor, implement the interdisciplinary terminology-based adaptive speech recognition method of any of the above embodiments.
[0182] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, random access memory (RAM), read-only memory (ROM), magnetic disks, or optical disks.
[0183] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, RAM, ROM, magnetic disks, or optical disks.
[0184] Corresponding to the computer storage medium described above, one embodiment also provides a computer device, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein the encoder executes the program to implement any of the interdisciplinary terminology-based adaptive speech recognition methods described in the above embodiments.
[0185] The aforementioned computer equipment utilizes multimodal context perception—that is, it integrates the dual-channel features of real-time speech context and structured PPT content—to drive subject-specific context determination, and then performs targeted speech recognition based on the corresponding subject-specific terminology database. This allows the equipment to improve recognition accuracy and enhance speech recognition performance in teaching scenarios by understanding the specific vocabulary of particular subjects.
[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0187] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An adaptive speech recognition method based on an interdisciplinary terminology database, characterized in that, include: Obtain a subject-specific hot term database, which includes: terms and subject tags corresponding to each term; Retrieve the recognized text from n seconds prior to the current time, and retrieve the title, bolded text, and full-page text of the PPT presentation at the current time. Based on the recognized text, text features are extracted through the text content processing module to generate the preceding speech text feature vector. Based on the titles, bold text, and full-page text of the PPT content, text features are extracted through the text content processing module to obtain the PPT structured text feature vector; Based on the aforementioned speech text feature vector and PPT structured text feature vector, output subject labels; Based on the subject tags, obtain the corresponding terms from the subject hot word library, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio; The process of extracting text features through the text content processing module includes: The input text is segmented using a pre-trained word segmentation model. Meaningless words in the stop word list are removed, and meaningful words are retained and recorded as the remaining terms. Based on the remaining terms, the weights of each remaining term are calculated using the TF-IDF values obtained from the statistics of a large-scale pre-training corpus and the word frequencies in the current text, which are dynamically calculated. Use a pre-trained word embedding model to convert each remaining term into a word vector; When extracting text features from the identified text, a text feature vector is generated based on the word vectors and according to the weights of each remaining term. When extracting text features from the title, bold text, and full-page text of the PPT content, a text feature vector is generated based on the word vectors, according to the weights of each remaining term, and by adding the weights of the title and bold text of the PPT content.
2. The interdisciplinary terminology-based adaptive speech recognition method according to claim 1, characterized in that, The process of calculating the weight of each retained term based on the retained terms, using the TF-IDF values obtained from large-scale pre-training corpus statistics and the dynamically calculated word frequencies in the current text, includes: The remaining terms are used to obtain TF-IDF value table v1 from large-scale corpus statistics, the importance score weights of manually annotated terms are obtained v2, and the word frequency of the remaining terms in the current text is dynamically calculated v3. These three are combined to obtain the final weight w of the term. w = v3 * (a * v2 + (1 - a) * v1); Where 'a' represents the preset subject preference weight.
3. The interdisciplinary terminology-based adaptive speech recognition method according to claim 1, characterized in that, Before the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the following is also included: For the remaining terms whose weights are higher than the preset term weight threshold, the word vector similarity is calculated based on the pre-trained word embedding model, and then K-Means clustering is used to select the center word of the largest cluster as the core keyword group. In the process of converting the remaining terms into word vectors using a pre-trained word embedding model, the core keyword groups are converted into word vectors using the pre-trained word embedding model.
4. The interdisciplinary terminology-based adaptive speech recognition method according to claim 1, characterized in that, The process of outputting subject labels based on the preceding speech text feature vector and the PPT structured text feature vector includes: The aforementioned speech text feature vector and PPT structured text feature vector are input into the subject orientation prediction model to determine the subject correlation and obtain the probability distribution of belonging to each subject. When the highest probability value of a subject is greater than the probability threshold, and the difference between the highest probability value and the second highest probability value is greater than the difference threshold, the subject corresponding to the highest probability value is output as the subject label. Here, the first probability threshold is greater than the second probability threshold. The subject-preference prediction model includes: The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, which are both of the first dimension. The feature fusion and compression are performed by an adversarial autoencoder to output a fused latent feature vector of the second dimension, where the second dimension is smaller than the first dimension. A deep semantic extraction layer is used to perform deep semantic modeling on the latent feature vectors using a multi-head self-attention mechanism combined with residual connections. The result prediction layer is used to calculate the probability distribution of each subject using a fully connected layer and a softmax function.
5. The interdisciplinary terminology-based adaptive speech recognition method according to claim 1, characterized in that, The process of obtaining the corresponding terms from the subject hot word library based on the subject tags, adding the obtained terms to the speech recognition decoding process, and outputting the speech recognition result of the current input audio includes: Based on the subject tags, retrieve the terminology list of the corresponding subject from the subject hot word database; Each term in the term list is converted into a word vector using a pre-trained word embedding model; Perform average pooling on all word vectors to generate a subject-specific hot word aggregation feature vector H; The speech recognition acoustic model processes the current audio frame and outputs a frame-level acoustic feature sequence A. The frame-level acoustic feature sequence A and the subject hot word aggregation feature vector H are passed through a multi-head cross-attention module to generate an enhanced acoustic feature E that incorporates hot word information; The enhanced acoustic feature E is input into the speech recognition decoder, and the final recognized text is output.
6. The interdisciplinary terminology-based adaptive speech recognition method according to claim 1, characterized in that, Also includes: Obtain a terminology mapping table related to a predefined discipline, wherein the Key in the terminology mapping table is a colloquial expression and the Value is a standardized written expression; The speech recognition results are corrected based on the terminology mapping table.
7. The interdisciplinary terminology-based adaptive speech recognition method according to claim 4, characterized in that, The preceding speech text feature vector is 128-dimensional, and the PPT structured text feature vector is 128-dimensional; The feature fusion and dimensionality reduction layer is used to concatenate the feature vectors of the preceding speech text and the PPT structured text, resulting in a 256-dimensional concatenation. The feature fusion and compression are performed through an adversarial autoencoder to output a fused 64-dimensional latent feature vector.
8. An adaptive speech recognition device with an interdisciplinary terminology database, characterized in that, include: The acquisition module is used to acquire a subject-specific hot term database, which includes: terms and their corresponding subject tags; acquire the recognized text from n seconds prior to the current time point; and acquire the title, bolded text, and full-page text of the PPT content at the current time point. The extraction module is used to extract text features based on the recognized text through the text content processing module, and generate the preceding speech text feature vector; based on the title, bold text and full-page text of the PPT content, the text content processing module extracts text features to obtain the PPT structured text feature vector. The determination module is used to output subject labels based on the feature vectors of the preceding speech text and the feature vectors of the PPT structured text; The recognition module is used to obtain the corresponding terms in the subject hot word library according to the subject tag, add the obtained terms to the speech recognition decoding process, and output the speech recognition result of the current input audio. The extraction module performs the process of extracting text features through the text content processing module, including: The input text is segmented using a pre-trained word segmentation model. Meaningless words in the stop word list are removed, and meaningful words are retained and recorded as the remaining terms. Based on the remaining terms, the weights of each remaining term are calculated using the TF-IDF values obtained from the statistics of a large-scale pre-training corpus and the word frequencies in the current text, which are dynamically calculated. Use a pre-trained word embedding model to convert each remaining term into a word vector; When extracting text features from the identified text, a text feature vector is generated based on the word vectors and according to the weights of each remaining term. When extracting text features from the title, bold text, and full-page text of the PPT content, a text feature vector is generated based on the word vectors, according to the weights of each remaining term, and by adding the weights of the title and bold text of the PPT content.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.