Speech recognition method for oral question and answer scenario and electronic device

Abstract

The application discloses a speech recognition method for oral question and answer scenarios and an electronic device. The method comprises the following steps: acquiring a hotword set corresponding to a current test question and an examinee's answering speech, and encoding the answering speech to obtain global acoustic features; performing multi-modal encoding on each hotword in the hotword set, respectively acquiring semantic features, phoneme pronunciation features and acoustic features, and fusing the three to obtain multi-modal hotword representation; inputting the global acoustic features and the multi-modal hotword representation into a speech decoder, and outputting recognized text by decoding with a bias attention mechanism. The application can effectively solve the problem of low recognition accuracy of homophonic confused words and low-resource out-of-vocabulary words in oral scenarios, and improve the robustness of the recognition result and the fairness of the downstream scoring.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, and more specifically, to a speech recognition method and electronic device for spoken question-and-answer scenarios. Background Technology

[0002] Existing automated oral assessment systems typically consist of an upstream speech recognition module and a downstream scoring module connected in series. The speech recognition module converts the candidate's spoken response into a text sequence, while the scoring module assesses the quality of the converted text. In recognition tasks targeting specific question scenarios, mainstream solutions generally employ context-biased strategies to enhance the capture of core keywords. These solutions typically use a separate bias encoder to vectorize relevant textual keywords, then utilize an attention mechanism to integrate the encoded text features into the speech recognition decoding process. This enhances the model's activation probability for specific words, thereby completing the conversion from acoustic signals to text content.

[0003] However, in complex real-world applications of spoken question answering, existing technologies still face persistent technical bottlenecks. Because spoken responses contain a large number of proper nouns, personal names, and specific terms, these words often have highly similar pronunciations or scarce training data. This makes it difficult for recognition systems to maintain high accuracy when dealing with homophonous words and out-of-vocabulary words with limited resources. For example, in English spoken language assessment scenarios, the proper noun for an athlete, "Su Yiming," is often misidentified by the model as the general word "Swimming" due to its similar pronunciation. This bias at the recognition level directly impacts the downstream scoring process, distorting the scoring results and affecting the overall reliability and fairness of the assessment system. This has become a key challenge restricting further technological development in this field. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a speech recognition method and electronic device for spoken question-and-answer scenarios, so as to improve the accuracy of hot word recognition and the overall accuracy of speech recognition. The specific solution is as follows:

[0005] Firstly, a speech recognition method for spoken question-and-answer scenarios is provided, including:

[0006] Obtain the set of hot words corresponding to the current test question and the candidate's answer voice, and encode the candidate's answer voice acoustic features to obtain global acoustic features;

[0007] Multimodal coding processing is performed on each hot word in the hot word set: the phoneme sequence and speech sample of the hot word are obtained, the hot word is text-encoded to obtain semantic features, the phoneme sequence is phoneme embedding-encoded to obtain phoneme pronunciation features, and the speech sample is acoustically encoded to obtain acoustic features;

[0008] The semantic features, phoneme pronunciation features, and acoustic features of the same hot word are fused to obtain a multimodal hot word representation of the hot word;

[0009] The global acoustic features and the multimodal hot word representations of each hot word are input into the speech decoder, which performs speech decoding through a biased attention mechanism and outputs the recognized text.

[0010] In one possible design, in another implementation of the first aspect of the embodiments of this application, before feature fusion of the semantic features, phoneme pronunciation features and acoustic features of the same hot word, the following is further included:

[0011] The phoneme pronunciation features of the hot words are used as query terms, and the acoustic features of the hot words are used as keys and values. Attention is calculated through a cross-attention mechanism to obtain target acoustic features that are aligned with the phoneme pronunciation features.

[0012] The process of fusing the semantic features, phoneme pronunciation features, and acoustic features of the same hot word specifically includes:

[0013] The semantic features, phoneme pronunciation features, and target acoustic features of the same hot word are fused together.

[0014] In one possible design, in another implementation of the first aspect of this application, the process of using the phoneme pronunciation features of the hot words as query terms, the acoustic features of the hot words as keys and values, and performing attention calculation through a cross-attention mechanism to obtain target acoustic features aligned with the phoneme pronunciation features includes:

[0015] Using each phoneme unit feature in the phoneme pronunciation features of the hot word as a query vector, and each acoustic frame feature in the acoustic features of the hot word as a key vector and a value vector, the attention weight between each phoneme unit feature and all the acoustic frame features is calculated.

[0016] The acoustic frame features are weighted and summed according to the attention weights to obtain the aligned acoustic sub-features corresponding to each phoneme unit, and the aligned acoustic sub-features corresponding to each phoneme unit constitute the target acoustic features.

[0017] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of text encoding the hot words to obtain semantic features includes:

[0018] The hot words are segmented into sub-words to obtain the sub-word unit sequence corresponding to the hot words;

[0019] Each of the sub-word units is mapped to a sub-word embedding vector to obtain the semantic features.

[0020] In one possible design, in another implementation of the first aspect of this application, the process of obtaining the voice samples of the hot words includes:

[0021] Based on the text content of the hot words, a synthesized speech signal corresponding to the hot words is generated through speech synthesis technology, which serves as the speech sample;

[0022] or,

[0023] Extract the speech samples corresponding to the hot words from the configured standard pronunciation lexicon.

[0024] In one possible design, in another implementation of the first aspect of the embodiments of this application, the process of fusing the semantic features, phoneme pronunciation features, and acoustic features of the same hot word to obtain a multimodal hot word representation includes:

[0025] The semantic features, phoneme pronunciation features, and acoustic features of the same hot word are concatenated along the feature dimension to obtain the concatenated feature;

[0026] The multimodal hot word representation is obtained by performing dimensionality reduction projection on the spliced ​​features through a learnable linear projection layer.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of this application, the speech decoder includes a speech attention branch and a bias attention branch;

[0028] The process of inputting the global acoustic features and the multimodal hot word representations of each hot word into a speech decoder, and then having the speech decoder perform speech decoding through a biased attention mechanism to output the recognized text includes:

[0029] Through the speech attention branch, attention is calculated using the current hidden state of the speech decoder as the query term and the global acoustic features as the key term to obtain a general decoding result;

[0030] Through the biased attention branch, attention is calculated using the current hidden state of the speech decoder as the query term and the multimodal hot word representation of each hot word as the key term, to obtain the biased enhanced decoding result;

[0031] The general decoding result and the bias-enhanced decoding result are probabilistically fused to output the recognized text at the current moment.

[0032] In one possible design, in another implementation of the first aspect of this application, the process of probabilistically fusing the general decoding result and the bias-enhanced decoding result to output the recognized text at the current moment includes:

[0033] By using a gated addition mechanism, the probability distributions corresponding to the general decoding result and the bias-enhanced decoding result are added and fused to obtain a fused probability distribution. The recognized text at the current moment is determined based on the fused probability distribution.

[0034] In one possible design, another implementation of the first aspect of the embodiments of this application further includes:

[0035] The identified text and the prior information of the current test question are input into the configured oral scoring model;

[0036] The oral scoring model is used to score oral responses based on the identified text and prior information of the current test question, and output the scoring results of the oral responses.

[0037] In a second aspect, an electronic device is provided, comprising: a memory and a processor;

[0038] The memory is used to store programs;

[0039] The processor is configured to execute the program to implement each step of the speech recognition method for spoken question-and-answer scenarios as described in any of the first aspects above.

[0040] By employing the aforementioned technical solution, this application introduces parallel encoding and fusion of text, phonemes, and acoustic dimensions, enabling multimodal hot word representations to not only include semantic information of hot words but also possess refined pronunciation rule constraints and realistic acoustic spectral features. This breaks through the information limitations of pure text single-modal representations and provides richer discrimination criteria for the decoding process. The combination of multimodal hot word representations and the decoder's biased attention mechanism effectively improves the ability to distinguish words with similar pronunciations during the decoding stage. This achieves high-precision recognition of proper nouns and easily confused words in complex spoken language scenarios, enhancing the robustness and stability of speech recognition results and ensuring the objectivity and accuracy of downstream spoken language scoring. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 A schematic diagram of an implementation system architecture for a speech recognition method for spoken question-and-answer scenarios provided in this application embodiment;

[0043] Figure 2 A schematic flowchart of a speech recognition method for spoken question-and-answer scenarios provided in this application embodiment;

[0044] Figure 3 This example illustrates an optional implementation flowchart of the speech recognition method for spoken question-and-answer scenarios in this application;

[0045] Figure 4 This application provides a schematic diagram of a speech recognition model structure and processing procedure.

[0046] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0049] In speech recognition schemes employing context bias strategies, the relevant technologies generally only encode the textual semantics of hot words, using the encoded textual features as bias information and integrating them into the decoder's process through an attention mechanism, in order to increase the activation probability of hot words in the speech recognition results.

[0050] However, the bias information in this scheme only includes single text modality features. In spoken question-and-answer scenarios (especially short question-and-answer scenarios), a large number of proper nouns, personal names, and competition terms have highly similar pronunciations and consistent stresses. Relying solely on text semantics cannot distinguish between homophones and confused words, which can easily lead to misidentification of homophones and near-homophones, resulting in errors in spoken language assessment.

[0051] In view of this, this application breaks through the input limitation of a single text modality and proposes a multimodal context-aware coding mechanism. It integrates three types of data: lexical information, phoneme features, and speech acoustic features of hot words to construct a multimodal hot word representation with high information density and strong robustness. This enhances the decoder's ability to bias and constrain hot words, and simultaneously improves the accuracy of hot word recognition and the overall speech recognition accuracy.

[0052] The method described in this application can be applied to oral question-and-answer scenarios, such as oral assessment scenarios for various languages.

[0053] This application provides a speech recognition method for spoken question-and-answer scenarios, which can be applied to, for example... Figure 1 The system architecture shown includes a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0054] Either terminal 100 or server 200 can be used independently to execute the speech recognition method for spoken question-and-answer scenarios provided in this application embodiment. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the speech recognition method for spoken question-and-answer scenarios provided in this application embodiment.

[0055] The following description Figure 1 The product form of the mid-terminal 100;

[0056] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, learning machine, voice recorder, teaching large screen, wearable device, conference terminal, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0057] This application provides a speech recognition method for spoken question-and-answer scenarios. Taking the application of this method to a computer device as an example, the computer device can specifically be... Figure 1The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The speech recognition method for spoken question-and-answer scenarios specifically includes the following steps:

[0058] Step S100: Obtain the set of hot words corresponding to the current test question and the candidate's answer voice, and perform acoustic feature encoding on the candidate's answer voice to obtain global acoustic features.

[0059] This step involves obtaining two types of input data sources: the original audio recordings of the examinee's spoken responses to the questions, i.e., the examinee's voice recordings, and the set of hot words contained in the current question.

[0060] Combination Figure 3 As shown, this application can parse exam papers. Exam papers contain structured data such as questions, answer keys, and keywords. By parsing the exam papers, the questions, answer keys, and keywords can be obtained.

[0061] Furthermore, hot word analysis can be performed on the test paper analysis results to extract all words from the test questions, reference answers, and question keywords, and after deduplication and word segmentation, generate a hot word set B for the current test question.

[0062] This step involves encoding the acoustic features of the candidate's speech to obtain global acoustic features. Specifically, this may include:

[0063] Standardized acoustic preprocessing is performed on the candidates' voice responses, such as... Figure 3 The diagram shows the extraction of acoustic features X, such as the Mel spectrum, using an acoustic feature extraction module. Based on this, the standardized acoustic features X of the examinee's speech are used as input to the speech recognition model, and fed into the speech encoder of the speech recognition model, as shown below. Figure 4 As shown, the speech encoder can be implemented using convolutional neural networks (CNNs) or a Conformer architecture. The input is the preprocessed acoustic features X, and the output is the global acoustic features containing speech temporal information and acoustic texture. These global acoustic features characterize the overall acoustic distribution of the examinee's spoken language, providing basic data support for subsequent decoding steps.

[0064] Step S110: Perform multimodal encoding processing on each hot word in the hot word set: obtain the phoneme sequence and speech sample of the hot word, perform text encoding on the hot word to obtain semantic features, perform phoneme embedding encoding on the phoneme sequence to obtain phoneme pronunciation features, and perform acoustic feature encoding on the speech sample to obtain acoustic features.

[0065] The process of performing multimodal encoding on each hot word in the hot word set can refer to extracting features from three dimensions—text, phonemes, and speech—in parallel for each hot word to construct a comprehensive description of the hot word.

[0066] Obtain the phoneme sequence of hot words to establish a mapping between word form and pronunciation. The phoneme sequence can be generated by querying a pre-set pronunciation dictionary or by using a grapheme-to-phoneme (G2P) model.

[0067] Obtaining speech samples of hot words is to acquire their standard acoustic representation. In this application, the speech samples corresponding to hot words can be obtained in various ways, such as, but not limited to: generating synthetic speech signals of hot words based on the text content of the hot words using text-to-speech technology; extracting real speech samples corresponding to the hot words from a configured standard pronunciation lexicon; or extracting pronunciation segments of the hot words from an existing speech corpus. Figure 3 The scheme shown only illustrates one implementation of obtaining speech samples of hot words through speech synthesis.

[0068] By employing speech synthesis technology to obtain speech samples of hot words, there is no need for manual collection and annotation of the speech of obscure proper nouns. The domain-adaptive speech synthesis TTS model can replicate the standard pronunciation and colloquial intonation of hot words in test questions, and generate simulated speech samples in batches that conform to the assessment scenario. This approach can supplement the acoustic data of unregistered hot words at low cost, and fundamentally solve the problem of recognition failure of newly added hot words caused by test question iteration.

[0069] It is understood that any speech sample that can provide acoustic pronunciation information of the hot word can be used to implement this application, and those skilled in the art can flexibly select one according to actual resource conditions and application requirements. In an optional embodiment, for cost control and ease of deployment, domain-adaptive speech synthesis (TTS) technology can be used to generate simulated speech samples of the hot word. In addition, when the hot word is a proper noun that appears frequently in the exam, a real person's pronunciation can be pre-recorded to obtain more natural acoustic features.

[0070] Text encoding of hot words yields semantic features, focusing on capturing the semantic information and morphological structure of vocabulary. This process can be achieved by directly embedding vectors into hot words to obtain their semantic features. Alternatively, hot words can be subword-tokenized to obtain a sequence of corresponding subword units. Each subword unit is then mapped to a subword embedding vector, which is then aggregated to form semantic features. Figure 4 As shown.

[0071] Sub-word segmentation aims to break down complete vocabulary into the smallest semantically meaningful language units, falling between characters and words. This process can be performed using pre-trained sub-word segmentation algorithms or statistical models to address the insufficient generalization ability of traditional integer word encoding methods when dealing with low-resource proper nouns or complex morphological vocabulary. For each hot word in the hot word set, the hot word string is converted into an ordered sequence of sub-word units based on a pre-defined sub-word list. This conversion is based on merging high-frequency substrings according to corpus statistical patterns, rather than simple character segmentation; for example, unhappy is segmented into un and happy, or the proper noun Yiming is segmented into Yi and ming. The generated sub-word unit sequence preserves the morphological structure information within the vocabulary, enabling the model to identify and share the features of different variant words under the same root, while significantly reducing the probability of out-of-vocabulary (OOV) words, thereby expanding the coverage of speech recognition models for diverse and personalized vocabulary in oral assessment scenarios.

[0072] By using a text encoder (such as an embedding layer matrix), discrete sequences of sub-word units are converted into vector representations in a continuous space, i.e., sub-word embedding vectors. All sub-word embedding vectors corresponding to the same hot word are combined sequentially to form the complete semantic features of that hot word. Through this fine-grained sub-word-level encoding, semantic features can flexibly express non-standard expressions such as word form variations and tone sandhi that may occur in spoken responses. Compared to coarse-grained word vectors, this representation method has stronger robustness and anti-interference ability in context-biased speech recognition tasks, providing precise semantic constraints for the decoder to accurately identify hot words in test questions.

[0073] Phoneme embedding encoding is performed on phoneme sequences to obtain phoneme pronunciation features, with an emphasis on incorporating linguistic pronunciation rules. This process, for example, involves inputting the phoneme sequence into a pre-trained phoneme embedding layer (which can be a text encoder), mapping each phoneme unit to a high-dimensional vector, thereby forming phoneme pronunciation features that contain pronunciation details.

[0074] Acoustic features are obtained by encoding acoustic features into speech samples, focusing on restoring the true acoustic properties of hot words. This process, for example, uses a speech encoder to convert speech samples into acoustic feature sequences, thereby obtaining acoustic features that reflect the spectral characteristics of standard pronunciation.

[0075] Step S120: The semantic features, phoneme pronunciation features and acoustic features of the same hot word are fused to obtain a multimodal hot word representation.

[0076] Compared to traditional solutions that only utilize single text modality information for context bias, such as Figure 4 As shown, this application can effectively compensate for the deficiencies of a single representation dimension and insufficient constraint robustness by integrating multimodal information.

[0077] One specific implementation of feature fusion is to concatenate the semantic features, phoneme pronunciation features, and acoustic features of the same hot word along the feature dimension to form a high-dimensional concatenated feature.

[0078] In some optional implementations, in order to adapt to the model dimensions and remove redundant information, such as Figure 4 As shown, the spliced ​​features can be dimensionality-reduced by a learnable linear projection layer to obtain the final multimodal hot word representation. This multimodal hot word representation incorporates prior knowledge from semantic, phonetic, and acoustic levels, significantly improving the model's ability to distinguish between low-resource hot words and homophonous words.

[0079] The learnable linear projection layer, as a fully connected neural network layer, contains trainable weight matrices and bias terms, enabling it to map high-dimensional concatenated features back to a predefined target feature dimension. During dimensionality reduction projection, this layer automatically learns the correlations between features of different modalities through training, preserving the key information that contributes most to the recognition task while suppressing noise and redundant information. The final output multimodal hot word representation has a unified dimension matching other modules of the model, inheriting multiple advantages of semantics, pronunciation, and acoustics, while also possessing good computational compatibility, allowing it to be effectively input into the speech decoder to execute the biased attention mechanism.

[0080] By combining the aforementioned feature dimension concatenation with dimensionality reduction projection, the transformation from discrete multimodal features to a unified high-order representation is achieved.

[0081] Step S130: Input the global acoustic features and the multimodal hot word representations of each hot word into the speech decoder. The speech decoder performs speech decoding through a biased attention mechanism and outputs the recognized text.

[0082] The speech decoder is configured to receive the global acoustic features output by the speech encoder as the basis for decoding, and to receive the multimodal hot word representations of each hot word as context bias information.

[0083] Speech decoding via a biased attention mechanism means that during the decoding process, the model not only focuses on general speech information in the global acoustic features, but also dynamically retrieves multimodal representations of hot words related to the current decoding state through the attention mechanism. The speech recognition decoder decodes the text content corresponding to the current speech segment based on the global acoustic features, and increases the probability of each hot word in the target hot word set in the probability distribution of the decoding output based on the bias information.

[0084] like Figure 4As shown, by introducing a biased attention branch, this speech decoder can enhance the output probability of hot words during decoding, while suppressing the activation probability of non-hot words or easily confused words.

[0085] The biased attention mechanism calculates attention weights by using the decoder's current hidden state as the query term and the multimodal hot word representations of each hot word as the key term. Based on this, it aggregates hot word information to obtain the bias-enhanced decoding result. Finally, the speech decoder outputs the recognized text based on the decoding result, which integrates global acoustic information and hot word bias information. This recognized text is the transcript of the examinee's speech, exhibiting higher hot word recognition accuracy.

[0086] The method provided in this embodiment utilizes multimodal prior information of hot words in test questions. Through multimodal coding fusion and biased attention decoding mechanisms, it effectively solves the recognition error problem caused by similar pronunciation of hot words and resource scarcity in spoken short question-and-answer scenarios. The multimodal hot word representation provides the decoder with rich semantic, phonetic, and acoustic constraints, enabling the speech recognition results to more accurately fit the context of the test questions, thereby providing a reliable data foundation for downstream spoken scoring.

[0087] The above embodiments of this application perform multimodal coding fusion on hot words, wherein:

[0088] The textual and phonemic information of hot words are both discrete symbolic descriptions of language, lacking the spectral characteristics of speech signals in physical acoustic space. For words like "Su Yiming" and "Swimming" with highly similar phonemic sequences, simply integrating the textual and phonemic information of hot words is still insufficient to establish a basis for differentiation from an acoustic physical dimension.

[0089] To this end, this application, based on the text and phoneme dual-modal scheme for hot words, further introduces the speech acoustic features of hot words as a third modality into the multimodal representation of hot words. The acoustic features of the speech samples of hot words belong to the continuous physical signal modality, forming a cross-domain information combination of discrete language symbols and speech signals with the text and phoneme two discrete symbol modalities. This enables the speech recognition model to obtain acoustic prior knowledge of "what the target hot word actually sounds like". There is a close synergy among the above three modalities. Text semantic features provide lexical identity anchoring, phoneme features provide structural decomposition at the phoneme unit level, and the acoustic features of speech samples provide actual speech spectrum information. The organic cooperation of the three enables the model to distinguish between confused words with similar pronunciations (or even similar phoneme sequences).

[0090] Furthermore, considering that existing single-modal context biasing schemes only achieve attentional interaction between hot word representations and global speech features at the text lexical level, without refining to the fine-grained matching dimension of phonemes and acoustic frames, and given that test takers' spoken responses generally exhibit natural speech distortions such as connected speech, elision, and uneven speech rate, coarse-grained alignment methods are unable to accurately match locally distorted acoustic segments. This easily leads to problems such as partial omission of target hot words and jitter in recognition results, and single-sentence response texts are prone to multiple fragmented recognition biases.

[0091] To this end, the method provided in this embodiment further constructs a phoneme-speech cross-attention mechanism, which interactively aligns the phoneme pronunciation features of hot words with the frame-level acoustic features of hot word speech samples one-to-one. It adapts to non-standard pronunciations such as connected speech, elision, and pauses in test takers' spoken language, and accurately applies contextual biases to locally distorted speech, eliminating fragmented omissions and misidentifications. Specifically, before feature fusion of the semantic features, phoneme pronunciation features, and acoustic features of the same hot word in the aforementioned step S120, the following steps are also included:

[0092] The phoneme pronunciation features of hot words are used as query terms, and the acoustic features of hot words are used as keys and values. Attention is calculated through a cross-attention mechanism to obtain target acoustic features that are aligned with the phoneme pronunciation features.

[0093] In this embodiment, to address the inherent time-scale mismatch between discrete phoneme units and continuous speech acoustic frames, a cross-attention mechanism is introduced to construct fine-grained alignment relationships. Phoneme articulation features represent the sequence of articulation units of hot words at the linguistic level, reflecting the syllable structure and phonetic representation of the vocabulary. Acoustic features, on the other hand, are derived from hot word speech samples and contain real spectral, pitch, and prosodic information.

[0094] This embodiment utilizes a cross-attention mechanism to establish a soft alignment mapping between phonemes and acoustic frames. During this calculation process, such as... Figure 4 As shown, phoneme pronunciation features serve as the query terms, initiating the query request and specifying which phoneme's corresponding acoustic information needs to be found. The acoustic features of hot words act as keys and values, forming the retrieval database. The keys are used to calculate the degree of matching with the query terms, and the values ​​are used to extract the actual feature content based on the matching degree. For each phoneme unit feature, its similarity score with all acoustic frame features is calculated. This score is then normalized to obtain the attention weight distribution, and these weights are used to perform a weighted summation of the acoustic frame features.

[0095] The logic for calculating this attention can be formally represented as:

[0096]

[0097] In the formula, Q represents the query matrix composed of phoneme pronunciation features, and K and V represent the key matrix and value matrix composed of the acoustic features of hot words, respectively. This represents the scaling factor for the dimension of the key vector, used to prevent the gradient from vanishing due to excessively large dot product values.

[0098] Through the above calculations, the model can automatically locate and focus on the corresponding acoustic segment in the synthesized speech for each phoneme unit, while ignoring interference from silent segments or other irrelevant phonemes. For example, for the hot word "Su Yiming," this mechanism can accurately anchor the phoneme "Yi" to an acoustic frame containing specific vowel formant features, thereby filtering out background noise or acoustic responses from other non-target phonemes. The final output is the target acoustic feature, which is no longer the original global acoustic feature, but a localized acoustic context representation that has been filtered by phoneme guidance and is highly aligned with the phoneme pronunciation features in time and content. This processing method effectively bridges the modal gap between text features and acoustic features, making the subsequent fused multimodal representations have a more rigorous internal logical consistency.

[0099] After obtaining the precisely aligned target acoustic features, the aforementioned step S120 is a process of feature fusion of the semantic features, phoneme pronunciation features, and acoustic features of the same hot word, specifically including:

[0100] The semantic features, phoneme pronunciation features, and target acoustic features of the same hot word are fused together.

[0101] In the feature fusion process, the semantic features, phoneme pronunciation features, and target acoustic features of the same hot word can be concatenated along the feature dimension to obtain a high-dimensional multimodal hot word representation that integrates three modal information. Its dimension is (N, 3D), where N represents the total number of all hot word tokens. D is the length of the feature vector output after single-modal encoding.

[0102] This embodiment of the method, through the fine-grained alignment and fusion of phonemes and acoustic features described above, solves the feature misalignment problem caused by relying solely on coarse-grained text and acoustic interaction in existing technologies. This method dynamically establishes a precise association between the phoneme level and the frame level through a cross-attention mechanism, enabling the acoustic details in the speech sample to be accurately assigned to the corresponding phoneme units. Even when faced with natural pronunciation distortions such as spoken linking and elision, it can still accurately match each phoneme unit of the target hot word with its corresponding acoustic segment. Thus, while preserving rich acoustic information, it eliminates background noise irrelevant to the current pronunciation. This synergistic effect effectively enhances the model's robustness to low-resource proper nouns and homophonous words, ensuring that contextual bias information plays a precise constraining role in the speech decoding process, thereby improving the accuracy and reliability of the entire speech recognition system.

[0103] Further, optionally, this embodiment describes the implementation method of using the phoneme pronunciation features of hot words as query terms, the acoustic features of hot words as keys and values, and performing attention calculation through a cross-attention mechanism to obtain target acoustic features aligned with the phoneme pronunciation features. Specifically, it may include:

[0104] S1. Using each phoneme unit feature in the phoneme pronunciation features of the hot word as the query vector, and each acoustic frame feature in the acoustic features of the hot word as the key vector and value vector, calculate the attention weight between each phoneme unit feature and all acoustic frame features.

[0105] Specifically, the smallest granular unit in phoneme pronunciation features—the phoneme unit feature—is used as the query initiator in the attention mechanism. The phoneme unit feature is used to characterize the pronunciation attributes of individual phonemes in hot words. The acoustic features of hot words consist of a series of consecutive acoustic frame features, each corresponding to the acoustic characteristics of the speech signal within a short time window.

[0106] A cross-attention mechanism is configured, treating the current phoneme unit feature as a query vector, responsible for retrieving the acoustic segment in the acoustic feature sequence that best matches the pronunciation of that phoneme. Each acoustic frame feature in the hot word acoustic feature sequence is simultaneously used as both a key vector and a value vector. A relevance score is obtained by calculating the dot product or scaled dot product between the query vector and each key vector. This score reflects the degree of matching between the current phoneme unit feature and the specific acoustic frame feature in terms of pronunciation or acoustic characteristics. Subsequently, all calculated relevance scores are normalized (e.g., using the Softmax function) to generate an attention weight distribution that corresponds one-to-one with each acoustic frame feature. This weight distribution dynamically focuses on frames that best reflect the acoustic characteristics of the current phoneme's pronunciation, while suppressing the influence of irrelevant or interfering acoustic frames.

[0107] S2. The features of all acoustic frames are weighted and summed according to the attention weights to obtain the aligned acoustic sub-features corresponding to each phoneme unit. The aligned acoustic sub-features corresponding to each phoneme unit constitute the target acoustic features.

[0108] Based on the attention weights calculated above, a weighted aggregation operation is performed on all acoustic frame features in the acoustic feature sequence. For each phoneme unit feature, its corresponding aligned acoustic sub-feature is generated by multiplying all acoustic frame features in the hot word acoustic features by their corresponding attention weights and then summing them up. This process essentially involves softly aligning discrete phoneme units with a continuous acoustic time axis.

[0109] By employing this weighted summation method, even if test-takers exhibit connected speech, elision, or changes in speaking speed that cause discrepancies between the actual phoneme duration and the standard pronunciation, the model can adaptively capture the most relevant acoustic segments from the original acoustic frame sequence through dynamic allocation of attention weights, thereby constructing robust aligned acoustic sub-features. Finally, the aligned acoustic sub-features corresponding to each phoneme unit within the hot word are arranged or recombined according to the original phoneme sequence order, thus forming a complete target acoustic feature aligned with the phoneme pronunciation features. This target acoustic feature serves as the foundational input for subsequent feature fusion, binding each phoneme unit with exclusive, finely aligned and optimized acoustic contextual information.

[0110] Reference Figure 4 This embodiment describes the structure of the speech recognition model.

[0111] A speech recognition model can include a speech encoder, a speech decoder, and a hot word context encoder.

[0112] Unlike traditional hot word context encoders that only encode the text modality of hot words, the hot word context encoder provided in this application can encode hot words from three modalities: text, phonemes, and speech. Specifically, it performs multimodal encoding processing on each hot word in the hot word set: it obtains the phoneme sequence and speech sample of the hot word, performs text encoding on the hot word to obtain semantic features, performs phoneme embedding encoding on the phoneme sequence to obtain phoneme pronunciation features, and performs acoustic feature encoding on the speech sample to obtain acoustic features.

[0113] Furthermore, speech recognition models can fuse semantic features, phoneme pronunciation features, and acoustic features of the same hot word to obtain a multimodal hot word representation. This process can involve concatenating the semantic features, phoneme pronunciation features, and acoustic features of the same hot word along the feature dimension to obtain a concatenated feature; then, using a learnable linear projection layer to perform dimensionality reduction projection on the concatenated feature to obtain the multimodal hot word representation.

[0114] The speech encoder in the speech recognition model can receive the acoustic information X of the input candidate's answer speech, such as the Mel spectrum, and further encode it to obtain global acoustic features containing speech temporal information and acoustic texture, which are then sent to the speech decoder.

[0115] The decoder receives in parallel the global acoustic features output by the speech encoder, the multimodal hot word representations output by the hot word context encoder, and the decoding output Y_(t-1) from the previous time step. It then performs general speech decoding, hot word bias enhancement, and probability fusion output in layers, ultimately generating the recognized text Y_t at the current time step (i.e., Figure 3 (Variable Y in the text).

[0116] In this embodiment, the speech decoder may include a speech attention branch and a bias attention branch (corresponding to respectively) Figure 4 The speech attention module and bias attention module in the text.

[0117] The process of a speech decoder decoding speech and outputting recognized text through a biased attention mechanism specifically includes:

[0118] By using the speech attention branch, attention is calculated with the current hidden state of the speech decoder as the query term and the global acoustic features as the key term, resulting in a general decoding result.

[0119] Specifically, a speech attention branch is used to retrieve general speech context information related to the current decoding state from global acoustic features. This branch is built on a multi-head attention mechanism, where the current hidden state of the speech decoder serves as the query term, encoding the generated historical text context and reflecting the model's semantic understanding of the current prediction position. Global acoustic features serve as key-value terms, containing a complete temporal acoustic representation of the input speech signal. By measuring the similarity between the current hidden state and each acoustic frame in the global acoustic features, attention weights are dynamically assigned to extract the acoustic segments most relevant to the current prediction target. This process allows the model to focus on the key regions that determine the current pronunciation and suppress interference from irrelevant noise or silence segments. The general decoding result obtained after weighted summation represents the text prediction probability distribution calculated based on the pure speech signal. This result is independent of the question-specific hot word constraints, thus ensuring the model's ability to recognize open vocabulary.

[0120] Furthermore, by using a biased attention branch, attention is calculated with the current hidden state of the speech decoder as the query term and the multimodal hot word representations of each hot word as the key term, thus obtaining the biased enhanced decoding result.

[0121] Specifically, hot word information specific to the test question is introduced to bias the decoding process. In this calculation, the current hidden state continues to serve as the query term, while the key-value terms are composed of the multimodal hot word representations of the aforementioned hot words. An attention mechanism is used to calculate the matching degree between the current decoding state and each candidate hot word. Given that the multimodal hot word representations contain rich pronunciation details and acoustic textures, this branch can accurately determine whether the current speech segment corresponds to a specific hot word in the test question. For example, when a test taker reads a sentence containing proper nouns, the biased attention branch increases the weight of the corresponding hot word in the output probability distribution based on acoustic similarity and semantic association. The resulting bias-enhanced decoding result is the probability distribution after hot word context bias correction, which focuses on improving the recognition confidence of the target keyword and effectively suppressing the probability of incorrectly selecting confusing words due to similar pronunciation (such as misidentifying Su Yiming as Swimming).

[0122] After obtaining the general decoding result and the bias-enhanced decoding result from the two branches, the general decoding result and the bias-enhanced decoding result can be probabilistically fused to output the recognized text at the current moment.

[0123] In other words, the decoding results obtained from the two branches are fused at the probabilistic level to generate the final recognized text. Among them, the general decoding result provides the prediction capability of a generalized language model in the open domain, ensuring the recognition accuracy of non-hot words; while the bias-enhanced decoding result provides the ability to finely discriminate hot words for specific test questions.

[0124] This fusion mechanism, while ensuring robustness in general recognition, specifically enhances the output probability of hot word regions. The speech recognition model can adaptively balance general speech patterns with specific contextual constraints. When outputting the recognized text at the current moment, it retains the fluency of natural language while significantly improving the recognition accuracy of key entities, providing a reliable text foundation for downstream spoken language scoring tasks.

[0125] Optionally, the process of probabilistically fusing the general decoding result and the bias-enhanced decoding result described above can be implemented by weighted summation or gating mechanisms, and the contribution of the two branches to the final result can be dynamically adjusted by learnable fusion weights.

[0126] Taking the gating mechanism as an example, the probability distributions corresponding to the general decoding result and the bias enhancement decoding result are added and fused through the gating addition mechanism to obtain the fused probability distribution. The recognized text at the current time is determined based on the fused probability distribution.

[0127] A gated addition mechanism is used to dynamically adjust the contributions of the general decoding branch and the bias enhancement branch in the final decision. This mechanism does not simply sum the outputs of the two branches numerically; instead, it introduces learnable weight parameters to adaptively weight the probability distributions corresponding to the general decoding result and the bias enhancement decoding result. The general decoding result corresponds to the base recognition probability output by the speech attention branch, reflecting the model's general understanding of global acoustic features; the bias enhancement decoding result corresponds to the hot word bias correction probability output by the bias attention branch, reflecting the reinforcement information for domain-specific vocabulary.

[0128] In its implementation, the gating addition mechanism first calculates a gating scalar or gating vector between 0 and 1. This gating value is dynamically generated by the model based on the context state at the current decoding moment and is used to determine the fusion ratio of the two branches. When the model determines that the current speech segment belongs to non-hot words in regular spoken content, the gating mechanism tends to suppress the weight of the bias enhancement branch, so that the fused probability distribution is mainly dominated by the general decoding result, thereby maintaining the model's universality and stability in open vocabulary recognition. Conversely, when the model detects that the current speech segment highly matches the acoustic features of the test hot words, the gating mechanism automatically increases the weight of the bias enhancement branch, shifting the fused probability distribution towards the hot words to achieve enhanced recognition of specific keywords. Through this dynamic adjustment mechanism, the fused probability distribution can balance the accuracy of general recognition and the targeting of hot word recognition, forming the final comprehensive decision basis.

[0129] The fused probability distribution represents the posterior probability of each candidate word in the vocabulary appearing at the current time. This distribution integrates the general information of global acoustic features and the bias information of multimodal hot word representations. Based on this, by searching for the candidate word with the highest probability value from the fused probability distribution, or by using decoding strategies such as beam search, the recognized text at the current time is finally locked and output. The resulting recognized text retains the semantic coherence of basic speech recognition and accurately hits the key hot words specified in the test question at critical moments, thus providing high-precision text input for downstream oral scoring tasks.

[0130] Through the aforementioned gating and addition mechanism, the general decoding result and the bias-enhanced decoding result are not statically mixed in a fixed ratio, but rather adaptively fused at the data stream level through learnable gating parameters. This allows the speech decoder to dynamically adjust its dependence on prior information about hot words based on the context, effectively correcting recognition deviations caused by similar pronunciations without sacrificing overall recognition fluency, and ensuring a balance between accuracy and robustness in the output recognized text.

[0131] In some embodiments of this application, after obtaining the recognized text corresponding to the examinee's speech, the recognized text can also be sent to a downstream oral scoring model, such as... Figure 3 As shown, this yields the scoring results for the oral responses. Specifically:

[0132] The identified text and prior information about the current test question are input into the configured oral scoring model.

[0133] The oral scoring model is used to score oral responses based on the identified text and prior information about the current test question, and output the score results of the oral responses.

[0134] The prior information of the current test question can refer to the structured test question data output by the test paper parsing module. This prior information includes, but is not limited to: the question stem text, the standard answer text, and a pre-defined set of keywords extracted from the question stem and standard answer. This prior information is extracted and formatted for storage during system initialization or test question loading, serving as the judgment criterion for the oral assessment model.

[0135] The oral assessment model is an automated evaluation module configured in the system backend to quantify and score candidates' oral performance based on multidimensional features. This model can employ an end-to-end neural network architecture or a hybrid architecture combining traditional statistical models and deep learning models. Internally, it includes a feature extraction layer, a similarity calculation layer, and a score regression layer.

[0136] The oral assessment model can comprehensively evaluate the completeness of the answer (recognized text), fluency of pronunciation, and degree of keyword matching, and output a standardized oral assessment score to represent the candidate's performance level on the current test question.

[0137] Because the method provided in the aforementioned embodiments can achieve high-precision identification of hot words, it blocks the transmission of noise introduced by identification errors to the scoring process. This embodiment can effectively avoid situations where candidates' compliant answers are mistakenly judged as invalid due to upstream errors such as misidentification of homophones, stabilize the standard deviation of question and answer scoring results, and thus ensure the fairness and consistency of oral assessment from the source, meeting the stringent requirements of standardized language tests for scoring reliability.

[0138] This application also provides an electronic device in its embodiments. (See reference...) Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, tablets, learning machines, large teaching screens, wearable devices, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2 or a program loaded from a storage device 8 into a random access memory (RAM) 3, to implement the speech recognition method for spoken question-and-answer scenarios of the foregoing embodiments of this application. When the electronic device is powered on, the RAM 3 also stores various programs and data required for the operation of the electronic device. The processing unit 1, ROM 2, and RAM 3 are interconnected via a bus 4. An input / output (I / O) interface 5 is also connected to the bus 4.

[0140] Typically, the following devices can be connected to I / O interface 5: input devices 6 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 7 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 8 including, for example, memory cards, hard drives, etc.; and communication devices 9. Communication device 9 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0141] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the speech recognition methods for spoken question-and-answer scenarios provided in this application.

[0142] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the speech recognition methods for spoken question-and-answer scenarios provided in this application.

[0143] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0145] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0146] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0147] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A speech recognition method for spoken question-and-answer scenarios, characterized in that, include: Obtain the set of hot words corresponding to the current test question and the candidate's answer voice, and encode the candidate's answer voice acoustic features to obtain global acoustic features; Multimodal coding processing is performed on each hot word in the hot word set: the phoneme sequence and speech sample of the hot word are obtained, the hot word is text-encoded to obtain semantic features, the phoneme sequence is phoneme embedding-encoded to obtain phoneme pronunciation features, and the speech sample is acoustic-feature-encoded to obtain acoustic features; The semantic features, phoneme pronunciation features, and acoustic features of the same hot word are fused to obtain a multimodal hot word representation of the hot word; The global acoustic features and the multimodal hot word representations of each hot word are input into the speech decoder, which performs speech decoding through a biased attention mechanism and outputs the recognized text.

2. The method according to claim 1, characterized in that, Before fusing the semantic features, phoneme pronunciation features, and acoustic features of the same hot word, the method further includes: The phoneme pronunciation features of the hot words are used as query terms, and the acoustic features of the hot words are used as keys and values. Attention is calculated through a cross-attention mechanism to obtain target acoustic features that are aligned with the phoneme pronunciation features. The process of fusing the semantic features, phoneme pronunciation features, and acoustic features of the same hot word specifically includes: The semantic features, phoneme pronunciation features, and target acoustic features of the same hot word are fused together.

3. The method according to claim 2, characterized in that, The process of using the phoneme pronunciation features of the hot words as query terms, the acoustic features of the hot words as keys and values, and performing attention calculation through a cross-attention mechanism to obtain target acoustic features aligned with the phoneme pronunciation features includes: Using each phoneme unit feature in the phoneme pronunciation features of the hot word as a query vector, and each acoustic frame feature in the acoustic features of the hot word as a key vector and a value vector, the attention weight between each phoneme unit feature and all the acoustic frame features is calculated. The acoustic frame features are weighted and summed according to the attention weights to obtain the aligned acoustic sub-features corresponding to each phoneme unit, and the aligned acoustic sub-features corresponding to each phoneme unit constitute the target acoustic features.

4. The method according to claim 1, characterized in that, The process of text encoding the hot words to obtain semantic features includes: The hot words are segmented into sub-words to obtain the sub-word unit sequence corresponding to the hot words; Each of the sub-word units is mapped to a sub-word embedding vector to obtain the semantic features.

5. The method according to claim 1, characterized in that, The process of obtaining speech samples of the hot words includes: Based on the text content of the hot words, a synthesized speech signal corresponding to the hot words is generated through speech synthesis technology, which serves as the speech sample; or, Extract the speech samples corresponding to the hot words from the configured standard pronunciation lexicon.

6. The method according to claim 1, characterized in that, The process of fusing the semantic features, phonemic features, and acoustic features of the same hot word to obtain a multimodal hot word representation includes: The semantic features, phoneme pronunciation features, and acoustic features of the same hot word are concatenated along the feature dimension to obtain the concatenated feature; The multimodal hot word representation is obtained by performing dimensionality reduction projection on the spliced ​​features through a learnable linear projection layer.

7. The method according to claim 1, characterized in that, The speech decoder includes a speech attention branch and a bias attention branch; The process of inputting the global acoustic features and the multimodal hot word representations of each hot word into a speech decoder, and then having the speech decoder perform speech decoding through a biased attention mechanism to output the recognized text includes: Through the speech attention branch, attention is calculated using the current hidden state of the speech decoder as the query term and the global acoustic features as the key term to obtain a general decoding result; Through the biased attention branch, attention is calculated using the current hidden state of the speech decoder as the query term and the multimodal hot word representation of each hot word as the key term, to obtain the biased enhanced decoding result; The general decoding result and the bias-enhanced decoding result are probabilistically fused to output the recognized text at the current moment.

8. The method according to claim 7, characterized in that, The process of probabilistically fusing the general decoding result with the bias-enhanced decoding result to output the recognized text at the current moment includes: By using a gated addition mechanism, the probability distributions corresponding to the general decoding result and the bias-enhanced decoding result are added and fused to obtain a fused probability distribution. The recognized text at the current moment is determined based on the fused probability distribution.

9. The method according to any one of claims 1-8, characterized in that, Also includes: The identified text and the prior information of the current test question are input into the configured oral scoring model; The oral scoring model is used to score oral responses based on the identified text and prior information of the current test question, and output the scoring results of the oral responses.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the speech recognition method for spoken question-and-answer scenarios as described in any one of claims 1 to 9.

Images