Speech recognition reliability optimization method, electronic device, and storage medium

Abstract

This application discloses a speech recognition reliability optimization method, electronic device, and storage medium. The method includes: acquiring a speech signal and extracting acoustic features from the speech signal; inputting the acoustic features into an automatic speech recognition model to obtain candidate transcribed text generated by the automatic speech recognition model; aligning the candidate transcribed text with a reference text and obtaining word-level alignment statistics based on the alignment results; calculating a utility value according to a preset utility calculation rule based on the word-level alignment statistics and a learnable penalty parameter; collecting human preference data for different candidate transcribed texts; fitting the penalty parameter according to the preference data so that the utility value aligns with the preference order reflected by the human preference data; using the aligned utility value as a reward signal for reinforcement learning and inputting it into the automatic speech recognition model for policy optimization; and outputting an optimized reliable transcription result through the policy-optimized automatic speech recognition model.

Description

Technical Field

[0001] This application belongs to the field of speech recognition technology, and in particular relates to speech recognition reliability optimization methods, electronic devices, and storage media. Background Technology

[0002] In related technologies, speech recognition systems typically train models through supervised learning to generate complete text transcription results, and then optimize them in the following ways: using WER (Word Error Rate) to measure recognition accuracy; forcing the output of complete text, even if the model is uncertain about some content; using token-level or sequence-level confidence to score the results; and some studies utilize reinforcement learning to optimize text quality or readability. All of these methods generally assume that the output must be a complete transcribed text.

[0003] The inventors discovered that related technologies suffer from one or more of the following shortcomings: 1. They cannot express the reliability of recognition results; 2. WER only measures the number of errors and cannot distinguish between them; 3. High-confidence errors; 4. Uncertain outputs that actively avoid errors; 5. Traditional ASR (Automatic Speech Recognition) tends to generate text that "seems complete but may be wrong"; 6. Confidence scores have a weak correlation with human real preferences; 7. Reinforcement learning optimization objectives often lack a reward definition consistent with human reliability perception. Summary of the Invention

[0004] This application provides a speech recognition reliability optimization method, electronic device, and storage medium to at least solve one of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a method for optimizing speech recognition reliability, comprising: acquiring a speech signal and extracting acoustic features from the speech signal; inputting the acoustic features into an automatic speech recognition model to acquire candidate transcribed text generated by the automatic speech recognition model, wherein the candidate transcribed text includes ordinary markers and placeholder markers; aligning the candidate transcribed text with a reference text and obtaining word-level alignment statistics based on the alignment results, wherein the reference text is manually annotated correctly transcribed text of the speech signal; calculating a utility value according to a preset utility calculation rule based on the word-level alignment statistics and a learnable penalty parameter, wherein the penalty parameter is used to adjust the penalty intensity for placeholder-related errors; collecting human preference data for different candidate transcribed texts; fitting the penalty parameter according to the preference data such that the calculated utility value aligns with the preference order reflected by the human preference data; using the aligned utility value as a reward signal for reinforcement learning and inputting it into the automatic speech recognition model to optimize the automatic speech recognition model's strategy; and outputting an optimized reliable transcription result through the strategy-optimized automatic speech recognition model.

[0006] Secondly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the speech recognition reliability optimization method of any embodiment of this application.

[0007] Thirdly, embodiments of this application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] The method in this application, by introducing placeholder tags, learnable penalty parameters, human preference alignment, and reinforcement learning optimization, enables the ASR model to strike an optimal balance between output completeness and output reliability, reducing misleading errors, improving human-perceived reliability, and outputting more credible text. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a method provided in one embodiment of this application; Figure 2 A comparison of the conventional assumptions provided for an embodiment of this application with the assumptions supporting the waiver mechanism; Figure 3 This embodiment of the present application shows the relationship between WER and RAS under different α settings on the LibriSpeech-test-clean dataset (as described in subsequent embodiments, using the Base+PH-Supv+RL model). Figure 4 A strategy for generating the first-stage yph provided in one embodiment of this application; Figure 5 This application provides an embodiment of running RAS on a noisy LibriSpeech dataset; Figure 6 Results provided for an embodiment of this application under clean (LibriSpeech) and code transformation (TALCS) conditions; Figure 7 Ablation study of a two-stage training process provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The inventors have found that one or more defects in the aforementioned related technologies are mainly caused by the following technical assumptions: 1. Existing evaluation metrics assume that complete output is better than partial output; 2. WER uses a uniform penalty for different types of errors; 3. It does not distinguish between "error output" and "active blanking"; 4. The reward or loss is not directly aligned with human preferences; 5. It lacks learnable reliability modeling parameters.

[0014] When faced with the aforementioned shortcomings, industry professionals commonly employ solutions including: 1. Increasing model size or training data; 2. Post-processing confidence filtering; 3. N-best reranking; 4. Calibration or uncertainty estimation. These methods share the common characteristic of still assuming that the complete text is the target output format.

[0015] The reason why the solution of this invention is not easy to think of is one or more of the following: 1. The goal of traditional ASR is to maximize the completeness of recognition, rather than reliability; 2. Treating "placeholder" as a reasonable output violates the traditional design concept; 3. It needs to be combined with: 4. Human preference modeling; 5. Utility function design; 6. Reinforcement learning optimization; and 7. It needs to establish a unified framework of human preference → mathematical utility (Utility Function) → RL (Reinforcement Learning) reward.

[0016] This application proposes a speech recognition reliability optimization method based on human preference alignment. Its core components include: 1. Allowing the recognition system to output partial transcriptions containing placeholders; 2. Constructing a utility function that distinguishes different error types; 3. Learning a penalty parameter α using human preference data; 4. Using the utility as a reinforcement learning reward; 5. Enabling the model to learn the optimal trade-off between "output completeness" and "reliability." Thus, this application can achieve one or more of the following beneficial effects: 1. Reducing misleading errors; 2. Improving human-perceived reliability; 3. Outputting more credible text.

[0017] Please refer to Figure 1 The diagram illustrates a system implementation flowchart of an embodiment of this application.

[0018] like Figure 1 As shown, the process steps are as follows: Step 1: Voice Input Processing The input speech signal is processed by an encoder to extract acoustic features.

[0019] Step 2: Generate candidate transcripts The ASR model generates text containing both regular tokens and placeholder tokens.

[0020] Step 3: Align Reference Text Match the hypothetical text with the reference text and perform statistical analysis: Correct word count: C; Placeholder-related errors (PEs); Common error NPE; Total word count N.

[0021] Step 4: Utility Calculation

[0022] Where α is a learnable parameter.

[0023] Step 5: Human Preference Collection Users can make preference choices for different transcriptions.

[0024] Step 6: Parameter Fitting The Bradley–Terry model is used to fit α, so that utility is consistent with human preferences.

[0025] Step 7: Reinforcement Learning Optimization Use utility as a reward to optimize the strategy of the ASR model.

[0026] Step 8: Output reliable transcription results In the process of implementing this application, the following alternative solutions were also explored: Option A: Delete low-confidence tokens based on a confidence threshold. Advantages: Simple to implement; no need for human preference data. Disadvantages: Thresholds are difficult to standardize; inconsistent with human perception of reliability; no global optimization objective.

[0027] Option B: Directly optimize WER. Advantages: Mature metrics; easy to implement. Disadvantages: Inability to distinguish error severity; encourages excessive text generation. This invention addresses these issues through utility modeling.

[0028] Between project initiation and final product development, the following beta versions were also produced: Beta version 1: α is manually set. Advantages: Simple to implement; allows for rapid concept validation. Disadvantages: Cannot adapt to different data domains. It deviates from human preferences.

[0029] Beta version 2: Supervised learning only for placeholder training, with only training labels modified. Advantages: Stable training. Disadvantages: Reliability metrics cannot be directly optimized. The final solution improves upon them through human preference learning.

[0030] The direct effects of this solution are improved ASR output reliability; reduced high-confidence errors; and significant alignment with human preferences. Furthermore, it can reduce misleading information in high-risk scenarios such as medical settings and conferences; increase the input credibility of downstream NLP systems; support risk-aware AI; achieve a controllable information output mechanism; and provide a general optimization framework for reliability-oriented AI.

[0031] Please refer to Figure 1 The diagram shows a flowchart of a speech recognition reliability optimization method provided in an embodiment of this application.

[0032] like Figure 1 As shown, in step 101, a speech signal is acquired, and acoustic features are extracted from the speech signal; In step 102, the acoustic features are input into an automatic speech recognition model to obtain candidate transcribed text generated by the automatic speech recognition model, wherein the candidate transcribed text includes ordinary markers and placeholder markers; In step 103, the candidate transcribed text is aligned with the reference text, and word-level alignment statistics are obtained based on the alignment results. The reference text is the manually annotated correct transcribed text of the speech signal. The reference text usually refers to the correctly annotated text (Ground Truth) corresponding to the speech signal, that is, the manually annotated transcription result that is considered to be completely correct, which is used to measure the error of the ASR model output. In step 104, based on the word-level alignment statistics and a learnable penalty parameter, the utility value is calculated according to a preset utility calculation rule, wherein the penalty parameter is used to adjust the penalty intensity for placeholder-related errors. In step 105, human preference data for different candidate transcribed texts is collected; In step 106, the penalty parameter is fitted based on the preference data so that the calculated utility value is aligned with the preference order reflected by the human preference data; In step 107, the aligned utility value is used as a reward signal for reinforcement learning and input into the automatic speech recognition model to optimize the strategy of the automatic speech recognition model. In step 108, the optimized automatic speech recognition model outputs an optimized reliable transcription result.

[0033] The method in this application embodiment, by introducing placeholder tags, learnable penalty parameters, human preference alignment, and reinforcement learning optimization, enables the ASR model to strike an optimal balance between output completeness and output reliability, reducing misleading errors, improving human-perceived reliability, and outputting more credible text.

[0034] In some optional embodiments, the word-level alignment statistics include the number of correct words, the number of placeholder-related errors, the number of ordinary errors, and the total number of words. The preset utility calculation rule includes: subtracting the number of placeholder-related errors (weighted by the penalty parameter) from the number of correct words, then subtracting the number of ordinary errors, and finally dividing by the total number of words, using the result as the utility value. Thus, through a specific utility calculation formula, the different severity levels of placeholder-related errors and ordinary errors can be quantitatively distinguished, allowing the utility value to more accurately reflect the differentiated impact of different error types on human perceived reliability.

[0035] In some optional embodiments, the placeholder marker is used to replace the regular text marker output when the automatic speech recognition model's confidence level for a speech segment is lower than a preset threshold, indicating that the recognition result is unreliable or should be left blank. This allows the ASR model to proactively output placeholders instead of forcibly outputting unreliable text, avoiding misleading users with high-confidence errors and improving the reliability of the recognition results.

[0036] In some optional embodiments, fitting the penalty parameter based on the preference data includes: using a pairwise comparison model, with the preference data as input, to construct a human preference probability distribution; iteratively adjusting the penalty parameter with the goal of maximizing the likelihood value of the preference data, until the ranking of the utility values ​​matches the ranking reflected by the human preference data. This allows the learning of the penalty parameter to be driven by human preference data, enabling the utility function to adapt to human reliability perception preferences in different application scenarios and avoiding biases caused by manually set parameters.

[0037] In some optional embodiments, the reward signal is used to encourage the automatic speech recognition model to balance output completeness and output reliability, wherein output completeness is reflected in maximizing the number of correct words, and output reliability is reflected in suppressing placeholder-related errors and general errors through the penalty parameter. Thus, the reward signal of reinforcement learning can guide the model to learn an optimal strategy, enabling the model to actively control the generation of unreliable outputs while pursuing recognition completeness, achieving a Pareto optimal tradeoff between the two.

[0038] In some optional embodiments, the penalty parameter is a real number greater than zero, and the method further includes: adaptively adjusting the value of the penalty parameter based on human preference data collected under different application scenarios to change the penalty intensity for placeholder-related errors. Thus, the penalty parameter can be adaptively adjusted according to the application scenario, enabling the method to flexibly adapt to the differentiated reliability requirements of different high-risk scenarios such as medical treatment, conferences, and autonomous driving.

[0039] In some optional embodiments, the method further includes: before outputting the reliable transcription result, determining whether the calculated utility value reaches a preset threshold; if the preset threshold is not reached, repeating the utility value calculation, human preference collection, penalty parameter fitting, and reinforcement learning optimization steps until the utility value reaches the preset threshold. Thus, through the threshold judgment and iterative optimization mechanism, the output transcription result can be ensured to meet the preset reliability standard, providing quantifiable quality assurance.

[0040] In some optional embodiments, the automatic speech recognition model is an end-to-end speech recognition model based on a deep neural network; during the reinforcement learning policy optimization process, the decoding strategy of the automatic speech recognition model is trained to include: for low-confidence acoustic features, actively outputting placeholder tags to replace conventional text prediction. Thus, the combination of the end-to-end model and the active placeholder output strategy enables the model to directly learn when to "leave blank" during the decoding stage, rather than filtering afterward, improving the global optimality of the decision.

[0041] In some optional embodiments, the human preference data is obtained through one or a combination of the following methods: by the user explicitly sorting or selecting from multiple candidate transcribed texts; or by implicitly extracting preference tags from the user's subsequent corrections to the transcription results. This supports both explicit and implicit preference collection methods, reducing data collection costs while improving the richness and usability of the preference data.

[0042] In some optional embodiments, the method is used for speech recognition in high-risk scenarios, including medical records, meeting minutes, and autonomous driving voice commands. In these high-risk scenarios, the risk cost of outputting unreliable text is configured to be higher than the risk cost of outputting text containing placeholders. Therefore, the solution of this application is particularly suitable for high-risk scenarios, reducing the spread of misleading information in medical, meeting, and autonomous driving scenarios, improving the input credibility of downstream NLP systems, and supporting the construction of risk-aware AI systems.

[0043] Automatic speech recognition (ASR) systems, under noisy or ambiguous conditions, often generate seemingly high-confidence but actually erroneous transcription results, potentially misleading users and downstream applications. Traditional standard evaluations based on word error rate (WER) focus only on accuracy and fail to characterize the reliability of transcription results. This paper proposes a transcription framework with abandonment awareness, enabling ASR models to display abandonment processing on uncertain segments. To evaluate the reliability of abandonment recognition, this paper further proposes a reliability-oriented index (RAS), which balances transcriptional information content with error avoidance, with its parameters calibrated by human preferences. For model training, this application first uses supervised bootstrapping for initialization. Subsequently, reinforcement learning is combined to train the ASR model with abandonment awareness. Experimental results show that the proposed method significantly improves the reliability of transcription results while maintaining competitive accuracy.

[0044] Modern automatic speech recognition (ASR) systems can achieve high accuracy in pristine acoustic environments, but in the presence of noise, speech overlap, signal degradation, or resource scarcity, they often still produce superficially fluent transcribed text. These outputs are typically the result of forced decoding based on weak acoustic evidence, producing errors that appear certain rather than explicitly demonstrating uncertainty. Such seemingly plausible but actually erroneous transcriptions can mislead subsequent decisions and, due to their readability, may further reduce the vigilance of human reviewers. This risk is particularly pronounced in high-risk applications with stringent transcription requirements, such as medical documents and legal records. However, mainstream evaluation paradigms do not adequately describe this failure mode.

[0045] A natural direction is to draw on reliability tools from the broader field of machine learning. While abstention (i.e., learning with a rejection option) and selective prediction have been shown to improve the reliability of machine learning by explicitly reducing risk at the cost of coverage, these methods typically model at the instance level, i.e., making a decision to accept or reject the entire input. This modeling approach is not suitable for Automatic Speech Recognition (ASR) because uncertainty in ASR is highly localized, and its utility depends on token / word-level decisions. Related research—ASR confidence estimation—offers a viable path to address this problem. However, most existing methods still follow a two-stage post-processing paradigm: they first generate the transcribed text and then append the uncertainty score as a separate layer of metadata. Therefore, confidence is implicitly modeled rather than integrated into the decoding process, resulting in a lack of internal mechanisms for actively removing unreliable segments during inference. Meanwhile, standard word error rate (WER) and its edit distance variants, as well as semantic metrics, implicitly assume that the model must generate the complete transcribed text. Therefore, these metrics only quantify the degree of difference between the hypothesis and the reference result, but fail to consider whether the system should output content (or how much content) when the evidence is insufficient. As a result, these metrics cannot assess whether a speech recognition system can make an appropriate trade-off between information content and reliability under conditions of uncertainty.

[0046] Figure 2 Comparison of traditional hypotheses and hypotheses supporting the abstention mechanism. The English and Chinese translations are as follows: GT: Reference text, hyp: Model transcription, Abstention-Aware hyp: Reliability-aware model transcription, May be misleading: Transcription results may mislead downstream tasks, Incomplete but reliable: Transcription results are incomplete but reliable.

[0047] In this application's embodiments, it is argued that a reliable Automatic Speech Recognition (ASR) system should be able to explicitly choose to abstain from predictions with local uncertainty. This application proposes a fine-grained abstention paradigm by expanding the output space by adding a dedicated placeholder, such as... Figure 2As shown. Unlike whole-sentence rejection, the method in this application allows the model to selectively discard ambiguous segments while performing highly deterministic transcription of the remaining content. This shift from passive scoring to an active, refined rejection model offers significant advantages for downstream applications: by providing clear, localized uncertainty signals, it prevents error propagation, ensuring that downstream modules perceive these placeholders as missing information rather than misleading illusions. To evaluate this paradigm, this application proposes a Reliability-Aware Score (RAS), a novel metric based on a modified edit distance that takes placeholder labeling into account. RAS achieves an explicit balance between transcriptional usability and error avoidance through tradeoff hyperparameters calibrated by human hearing tests. This provides a theoretically sound objective for the evaluation and optimization of Reliability-Aware ASR.

[0048] This application further validates the practical application value of the framework by developing a Whisper model that supports abstention decisions. By employing a two-stage training process—including supervised guided learning followed by reinforcement learning (RL) based on RAS as reward—this application significantly improves system reliability. Experimental results show that the method in this application greatly improves the reliability of transcription results while maintaining competitive accuracy, especially in resource-scarce and noisy environments. The main contributions of this application are summarized as follows: This application extends selective prediction to sequential ASR, enabling fragment-level abstention rather than whole-sentence rejection.

[0049] This application proposes RAS, a reliability-oriented index calibrated based on human preference data, used to quantify the trade-off between information content and error-avoidance tendencies.

[0050] This application establishes a robust training pipeline that combines supervised learning and reinforcement learning, significantly improving the reliability of ASR models in harsh acoustic environments.

[0051] Reliability rating To enable the Automatic Speech Recognition (ASR) model to have explicit rejection capability, this application adds a special placeholder PH to the original vocabulary. Unlike ordinary words, PH stands for "abstention": when the model encounters low-quality or ambiguous speech and cannot reliably determine its underlying content, it outputs PH to represent uncertainty, rather than giving a potentially misleading guess. With the introduction of PH, this application modifies the standard WER to RAS (an metric that considers the rejection situation), which is calculated using a dynamic programming formula that extends the standard definition of edit distance.

[0052] Dynamic programming model Suppose a reference text `ref` of length N > 0 contains no hyphens (PHs), while a hypothetical text `hyp` of length M may contain PHs. Let `ri` and `hj` represent the `i`-th and `j`-th words (counted from 0) in `ref` and `hyp`, respectively. Unlike standard edit distance, a single PH may be aligned with zero or more consecutive reference words, reflecting the omission of uncertain acoustic segments. Consecutive PHs are merged to avoid redundancy.

[0053] Treating errors introduced by PH (Programming Physics) the same as ordinary word errors when expanding the definition of edit distance is inappropriate. While PH fails to provide useful information, it does not assert an incorrect lexical term, thus avoiding the introduction of potentially misleading content. In this sense, errors associated with PH are relatively less harmful and maintain greater reliability. To reflect this distinction, this application assigns a weighting factor α∈(0, 1) to all PH-related operations. Therefore, this application will... α i,j Defined as the minimum weighted edit distance from ref[0:i) to hyp[0:j) under an alignment scheme that considers the abstention mechanism, its boundary conditions are:

[0054] The transition is defined as the minimum of the three potential costs:

[0055] in

[0056] When h j-1 When = PH, this corresponds to the standard editing operation, but introduces a flexible comparison method for PH. When h j-1 When PH = , the replacement term Sub allows PH to be compared with fragments r[k:i) of arbitrary length in the reference text, thus effectively “absorbing” i - k deletion operations at a discounted cost of α per word. This corresponds to a many-to-one mapping where PH covers a continuous range of uncertain content. When PH does not appear in hyp, the formula simplifies precisely to the standard edit distance. This application further defines the weighted edit distance from ref to hyp as g. α (ref, hyp) := g α The computation time complexity of N and M is O(N 2M).

[0057] Figure 3The relationship between WER and RAS under different α settings on the LibriSpeech-test-clean dataset (using the Base+PH-Supv+RL model, as described in subsequent examples). Each data point corresponds to one speech sample.

[0058] Beyond WER: The Definition of RAS From the optimal dynamic programming alignment results, this application obtained the correct matching counts C(ref, hyp) and g. α (ref, hyp). Inspired by existing reliability-aware decision utility models in the field of Natural Language Processing (NLP), this application defines the reliability assessment metric RAS R(ref, hyp) as:

[0059] In this context, utility measures the proportion of correctly transcribed words, while cost, as described in Section 2.1, penalizes alignment errors through difference weighting.

[0060] Where α ∈ (0, 1). This formula directly balances the relationship between information content and false commitments within the proposed alignment framework. Its maximum value is 1; a higher RAS value indicates a more reliable and useful ASR model. When PH does not appear in the hyp, the RAS value simplifies to a form similar to the standard WER:

[0061] Both RAS and WER are monotonic functions of the total number of word errors (S + D + I). Therefore, in the absence of PHs, from an optimization perspective, maximizing RAS is equivalent to minimizing WER. Figure 3 As shown, there is a clear negative correlation between the proposed objective function and WER.

[0062] In practical applications, α should be calibrated based on human preferences for reliability in specific application scenarios, rather than being selected based on experience.

[0063] Alignment of human preferences through auditory testing The hearing test was conducted using the BeaqleJS framework. For each audio sample i, participants were shown the actual transcribed text (G). i ) and two simulated ASR outputs: one is the regular transcribed text without PH (A i The other is the transcribed text that takes into account the "abstention" situation (B). i Transcribed text B iThe construction method is: to use A i In the transcription, incorrect words were replaced with pseudonyms (PHs), and a small subset of correct words were additionally masked to simulate conservative abstention. Participants were asked to choose the transcribed text they considered more reliable, while also being given the option of "cannot decide".

[0064] This application estimates the value of α based on collected user preferences. Let the number of audio files be K. For each audio file i, let k... i A k i B and k i C Let Ai represent the preference for regular transcription, Bi represent the preference for transcription considering abstention, and si represent the number of subjects with no preference, where si = k. i A + k i B + k i C Let P i Indicates human preference for audio (i). i The probabilities R, C, gα, and N have the same meanings as in the previous embodiments. This application uses notation... M i := M(G i B i ) - M(G i A i ), where M ∈ {R, C, g α Following the Bradley–Terry model, P i Modeled as R i The function. Note R(G). i A i ) and R(G i B i They have the same denominator:

[0065] Where σ(·) represents the logistic function. This application defines the preference likelihood loss and incorporates indifferent responses as a regularization term to encourage RAS differences to be as small as possible. The final objective function is the weighted sum of these two:

[0066] Here, λ controls the strength of the indifference constraint. This formulation allows for robust calibration of α, thereby aligning the proposed RAS with human reliability preferences.

[0067] Figure 4The strategy for generating the first-stage yph. Mispredicted segments will be replaced with PH.

[0068] Training an ASR model with the ability to abstain The training process of this application is divided into two consecutive stages, which aims to enhance the model's ability to perceive uncertainty and its self-correction ability.

[0069] Phase 1: Placeholder Supervision (PH-Supv) The main goal of this phase is to build a dataset to guide the base model in identifying and labeling prediction errors.

[0070] Training set construction Given an audio input 'a' and its ground truth transcribed text 'ygt', this application first uses a base model M to perform inference and generate a hypothesis 'yhyp'. To locate recognition errors, this application uses a standard word error rate (WER) calculation method to compare 'yhyp' with 'ygt'.

[0071] This alignment process generates a series of operations, each belonging to one of {Match, Substitute, Insert, Delete}. This application defines a ground truth-guided PH replacement sequence yph composed of these operations, constructed as follows: Match: The corresponding text in yhyp is preserved in yph.

[0072] Replacement / Insertion: In yhyp, err text fragments are replaced with special tokens PH. To maintain the relative sequence length, the number of PH tokens is determined by the tokenizer T of the base model. Specifically, this application inserts N PH tokens, where N = |T(terr)|.

[0073] Deletion: In this case, yhyp omits the text present in ygt. This application inserts a PH token into yph to correspond to the missing text fragment tmiss in ygt. Similar to the above case, the number of PH tokens is set to N = |T(tmiss)|.

[0074] Figure 4 An example of the above process is shown. This process ultimately generates a refined training set consisting of (a, yph) pairs.

[0075] Model training Before training, this application expands the vocabulary of the base model (Base) M to include the new token PH and ensures that the tokenizer treats it as a single, indivisible token. Subsequently, the model is fine-tuned on the constructed dataset using the same ASR objective function as M. In this embodiment, Whisper is used as the base model M, therefore the cross-entropy objective function is used at this stage.

[0076] Phase 2: Group Relative Strategy Optimization (RL) Once the model has the ability to output a PH, this application employs the Group Relative Policy Optimization (GRPO) algorithm to optimize its output policy, using the statement-level RAS proposed in the aforementioned embodiments as the reward signal. Specifically, for each input cue word q, GRPO will optimize the current policy π. θold A set of G outputs {o1, o2, ..., o...} are sampled. G Each output o i All rewards are evaluated using the RAS reward function to obtain the reward r. i Subsequently, the relative advantage of the group is calculated as follows:

[0077] This strategy achieves optimization by maximizing the following objective function:

[0078] in The importance sampling ratio, To truncate the parameter, D i,t KL Represents relative to the reference strategy π ref The KL divergence of each token is defined in the relevant technical documentation and will not be elaborated here.

[0079] experiment Dataset This application conducted experiments on two datasets: LibriSpeech (a widely used English audiobook corpus) and the TALCS corpus (an English-Mandarin code-switching dataset). For LibriSpeech, this application used train-clean-360 for training and test-clean for evaluation. To evaluate the reliability of ASR under harsh acoustic conditions, this application simulated a noisy variant (Noisy LibriSpeech) by injecting additive white Gaussian noise (AWGN) throughout the entire duration of the original audio samples. This application generated four distinct training and test subsets with fixed signal-to-noise ratios (SNR) of {0, 5, 10, 20} dB. For all other datasets, this application followed the official training, development, and testing partitions.

[0080] Evaluation metrics aligned with human results This application uses RAS (defined in the foregoing embodiments) as the evaluation criterion, where the α value is estimated through a manually aligned hearing test. The test design follows the foregoing embodiments. Audio samples were manually selected from the “Medical ASR Recording Dataset” (173 samples) and the “AMI Corpus” (191 samples), covering terminology-intensive medical speech and noise-affected conversational speech. As previously stated, two transcriptional variants were constructed for each sample. A complete list of samples and transcription pairs can be found in the supplementary materials.

[0081] After validity screening, a total of 980 preference annotations were collected from 42 participants. The upper bound of the manual or acle, calculated by majority voting and consistency between annotators and consensus, reached 78.11%, indicating strong consistency among annotators. When λ = 0.1 (corresponding to a low draw rate of 6.63%), minimizing the objective function in equation (12) yields α = 0.5064 and the average RAS difference. U = 0.0461, which indicates that the proposed RAS framework is consistent with human reliability judgments.

[0082] Figure 5 : Run RAS on the noisy LibriSpeech dataset.

[0083] Experimental setup This application uses the following experimental notation: Base & Base+Logit: This application uses Whisper-Tiny as the baseline (Base). Base+Logit employs a logit-based PH replacement strategy, aggregating token-level confidence (word-level for English; character / word-level for Chinese) through multiplication, following the official heuristic. For each dataset, this application replaces tokens below the confidence threshold with PH, and adjusts the threshold (typically within the range of [0.1, 0.3]) to maximize RAS.

[0084] Base+PH-Supv: This stage utilizes the PH replacement generated from the Base prediction results on the training set (see the aforementioned embodiment for details). This application uses AdamW

[30] to fine-tune the decoder and embedding for 8 epochs, with a batch size of 64, a learning rate of 1.0 × 10-5, a warm-up of 1,000 steps, and linear decay.

[0085] Base+PH-Supv+RL: Based on Base+PH-Supv initialization, this application further optimizes PH supervision through GRPO. The decoder / embedding layer is unfrozen and trained with a global batch size of 64. For each cue, G = 8 responses are sampled using a temperature of 0.7 and a top-p of 0.95 (512 samples per step). This application employs an adaptive KL penalty term with an initial value β0 = 0.2, updated every 50 steps: β t+1 = β t exp(η(KL t - KL target ), where η = 0.02, KL target = 30. This application uses the Adam optimizer with a peak learning rate of 2×10⁻⁶ (1,000-step warm-up and linear decay) and early stopping based on the stabilization of the reward mean.

[0086] GT-guided PH replacement: This application applies the algorithm in the foregoing embodiments to the Base output and constructs a PH replacement sequence under the guidance of the real labels in the test set.

[0087] Main results Figure 6 Performance of RAS on the LibriSpeech and TALCS datasets. The best results for each dataset are highlighted in bold, excluding the ground truth-based PH replacement system.

[0088] Figure 6Results are reported under clean (LibriSpeech) and code translation (TALCS) conditions. Our proposed method, Base+PH-Supv+RL, consistently outperforms the two baseline models across all metrics. On the TALCS dataset, the base model has a negative RAS (-0.11), while our method achieves 0.48, demonstrating the effectiveness of combining PH-supervised pre-training with reinforcement learning (RL) in challenging code translation scenarios. The baseline (Base+Logit) using only logods yields only a slight improvement, confirming that simple confidence thresholding is insufficient. The ground-truth-guided PH-supplementation method provides an upper bound approaching oracle-like accuracy.

[0089] Figure 5 The performance under noisy conditions (Noisy LibriSpeech) is reported. At a signal-to-noise ratio (SNR) of 0 dB, the proposed method improves the Resonant Achievement Score (RAS) by 0.2657 compared to the baseline model, while the improvement is only 0.0208 under noise-free conditions, highlighting its robustness advantage in high-noise environments. The RAS improvement brought by the proposed method becomes increasingly significant as the SNR decreases, indicating that the proposed training procedure (i.e., PH-Supv+RL) can deliver even greater robustness improvements in more challenging acoustic environments.

[0090] Ablation Research Figure 7 Ablation study of two-stage training process.

[0091] Figure 7 This paper presents a stripping experiment designed to isolate the contributions of GRPO-based reinforcement learning training. By comparing Base+PH-Supv and Base+PH-Supv+RL, this application finds that reinforcement learning consistently improves RAS and utility metrics on both datasets. On the TALCS dataset, although the cost metric increases slightly, this reflects the optimization of RL towards RAS: the model learns that the usefulness gains from predicting more words outweigh the cost incurred by the incorrect decoding of tokens partially replaced by PH. The net effect is a significant improvement in RAS, which confirms that the GRPO-based RL stage provides complementary benefits on top of supervised placeholder training.

[0092] It is worth noting that on the TALCS dataset, Figure 7 Both Base+PH-Supv and Base+PH-Supv+RL in the model surpass [the performance of the previous model]. Figure 6The upper bound of PH replacement based on ground truth (GT) is discussed. Since the Base model performs poorly here, GT-based replacement is fundamentally limited by its weak code transformation capabilities. In contrast, PH-Supv training truly improves model performance: even without complete GT-transcribed text, the correct tokens retained after replacing Base errors provide sufficient supervision, significantly improving TALCS performance, and RL further amplifies this advantage.

[0093] This application redefines the reliability of Automatic Speech Recognition (ASR) by introducing a placeholder-based abstention mechanism, shifting the paradigm from speculative transcription to risk-aware reporting. It proposes the Risk-Aware Assessment (RAS) framework, a principle-based metric calibrated based on human preferences, designed to balance information richness with error avoidance. By implementing a training process that incorporates the abstention mechanism, this application demonstrates that the model can significantly improve reliability without sacrificing competitive accuracy. This framework establishes a new standard for trusted speech processing. This application anticipates that the RAS framework will provide a solid evaluation foundation for future trusted speech processing research, particularly as a reliable reward signal for optimizing model behavior in reinforcement learning.

[0094] In other embodiments, this application also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the speech recognition reliability optimization method in any of the above method embodiments for use in draft models and target verification models; As one implementation, the non-volatile computer storage medium of this application stores computer-executable instructions, which are configured as follows: Acquire speech signals and extract acoustic features from the speech signals; The acoustic features are input into an automatic speech recognition model to obtain candidate transcribed text generated by the automatic speech recognition model, wherein the candidate transcribed text includes ordinary markers and placeholder markers; The candidate transcribed text is aligned with the reference text, and word-level alignment statistics are obtained based on the alignment results, wherein the reference text is the manually annotated correct transcribed text of the speech signal; Based on the word-level alignment statistics and a learnable penalty parameter, the utility value is calculated according to a preset utility calculation rule, wherein the penalty parameter is used to adjust the penalty intensity for placeholder-related errors. Collect human preference data for different candidate transcribed texts; The penalty parameter is fitted based on the preference data so that the calculated utility value is aligned with the preference order reflected by the human preference data; The aligned utility value is used as a reward signal for reinforcement learning and input into the automatic speech recognition model to optimize the policy of the automatic speech recognition model. The optimized automatic speech recognition model outputs a reliable transcription result.

[0095] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the speech recognition reliability optimization method and the system. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to the processor, and these remote memories may be connected to the speech recognition reliability optimization method via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] This application also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described speech recognition reliability optimization methods.

[0097] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the device includes one or more processors 810 and a memory 820. Figure 8 Taking a processor 810 as an example, the device for optimizing speech recognition reliability and system may further include an input device 830 and an output device 840. The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking a bus connection as an example, the memory 820 is the aforementioned non-volatile computer-readable storage medium. The processor 810 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the speech recognition reliability optimization method described in the above embodiment. The input device 830 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the large language model routing device. The output device 840 may include a display device such as a screen.

[0098] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0099] In one implementation, the above-described electronic device is applied in a large-scale language model routing device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire speech signals and extract acoustic features from the speech signals; The acoustic features are input into an automatic speech recognition model to obtain candidate transcribed text generated by the automatic speech recognition model, wherein the candidate transcribed text includes ordinary markers and placeholder markers; The candidate transcribed text is aligned with the reference text, and word-level alignment statistics are obtained based on the alignment results, wherein the reference text is the manually annotated correct transcribed text of the speech signal; Based on the word-level alignment statistics and a learnable penalty parameter, the utility value is calculated according to a preset utility calculation rule, wherein the penalty parameter is used to adjust the penalty intensity for placeholder-related errors. Collect human preference data for different candidate transcribed texts; The penalty parameter is fitted based on the preference data so that the calculated utility value is aligned with the preference order reflected by the human preference data; The aligned utility value is used as a reward signal for reinforcement learning and input into the automatic speech recognition model to optimize the policy of the automatic speech recognition model. The optimized automatic speech recognition model outputs a reliable transcription result.

[0100] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones, multimedia phones, feature phones, and low-end phones, etc.

[0101] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, etc.

[0102] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0103] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0104] (5) Other electronic devices with data interaction functions.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for optimizing the reliability of speech recognition, comprising: Acquire speech signals and extract acoustic features from the speech signals; The acoustic features are input into an automatic speech recognition model to obtain candidate transcribed text generated by the automatic speech recognition model, wherein the candidate transcribed text includes ordinary markers and placeholder markers; The candidate transcribed text is aligned with the reference text, and word-level alignment statistics are obtained based on the alignment results, wherein the reference text is the manually annotated correct transcribed text of the speech signal; Based on the word-level alignment statistics and a learnable penalty parameter, the utility value is calculated according to a preset utility calculation rule, wherein the penalty parameter is used to adjust the penalty intensity for placeholder-related errors. Collect human preference data for different candidate transcribed texts; The penalty parameter is fitted based on the preference data so that the calculated utility value is aligned with the preference order reflected by the human preference data; The aligned utility value is used as a reward signal for reinforcement learning and input into the automatic speech recognition model to optimize the policy of the automatic speech recognition model. The optimized automatic speech recognition model outputs a reliable transcription result.

2. The method according to claim 1, characterized in that, The word-level alignment statistics include the number of correct words, the number of placeholder-related errors, the number of ordinary errors, and the total number of words. The preset utility calculation rule includes: subtracting the number of placeholder-related errors (weighted by the penalty parameter) from the number of correct words, then subtracting the number of ordinary errors, and finally dividing by the total number of words, and using the calculation result as the utility value.

3. The method according to claim 1, characterized in that, The placeholder marker is used to replace the regular text marker output when the confidence level of the automatic speech recognition model for a speech segment is lower than a preset threshold, so as to indicate that the recognition result is unreliable or needs to be left blank.

4. The method according to claim 1, characterized in that, The fitting of the penalty parameter based on the preference data includes: A pairwise comparison model is used, with the preference data as input, to construct a probability distribution of human preferences; With the goal of maximizing the likelihood of the preference data, the penalty parameter is iteratively adjusted until the ranking of the utility values ​​is consistent with the ranking reflected by the human preference data.

5. The method according to claim 1, characterized in that, The reward signal is used to encourage the automatic speech recognition model to make a trade-off between output completeness and output reliability, wherein output completeness is reflected in maximizing the number of correct words, and output reliability is reflected in suppressing placeholder-related errors and general errors through the penalty parameter.

6. The method according to claim 1, characterized in that, The penalty parameter is a real number greater than zero, and the method further includes: The value of the penalty parameter is adaptively adjusted based on human preference data collected in different application scenarios to change the penalty intensity for placeholder-related errors.

7. The method according to claim 1, characterized in that, The method further includes: Before outputting the reliable transcription results, it is determined whether the calculated utility value reaches a preset threshold; If the preset threshold is not reached, the utility value calculation, human preference collection, penalty parameter fitting, and reinforcement learning optimization steps are repeated until the utility value reaches the preset threshold.

8. The method according to claim 1, characterized in that, The automatic speech recognition model is an end-to-end speech recognition model based on a deep neural network; during the strategy optimization process of the reinforcement learning, the decoding strategy of the automatic speech recognition model is trained to include: for acoustic features with low confidence, actively outputting placeholder tags to replace conventional text prediction.

9. The method according to claim 1, characterized in that, The human preference data is obtained through one or a combination of the following methods: by collecting data after the user explicitly sorts or selects one of two candidate transcribed texts; or by implicitly extracting preference tags from the user's subsequent modification behavior of the transcription results.

10. The method according to claim 1, characterized in that, The method is used for speech recognition in high-risk scenarios, including medical records, meeting minutes, and autonomous driving voice commands. In these high-risk scenarios, the risk cost of outputting unreliable text is configured to be higher than the risk cost of outputting text containing placeholders.

Images