A method for multilingual speech recognition model construction

Abstract

The application discloses a kind of for multilingual speech recognition model construction method, it is related to computer technology field, the present application generates candidate text in parallel through oral branch and lyrics branch, and it is compared under the same oral language model to the candidate in window and carries out fluency evaluation, utilize gate determination and mark the time window that will be more in line with lyrics generation rule as inhibition interval, to avoid the pollution of lyrics semantics to oral subtitle at output end;In addition, map the difference of gate as continuous gate intensity, and the intensity is applied to the word list output in decoding process to form soft inhibition, so that non-oral segment is more inclined to output placeholder rather than random oral lexical unit, while introducing hysteresis or continuous time window consistent trigger constraint, reduce the gate jitter at the boundary of overlapping time window and improve subtitle coherence.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for constructing multilingual speech recognition models. Background Technology

[0002] In the creation and dissemination of short video content on social media, platforms typically provide automatic captioning capabilities to improve accessibility and retrieval efficiency. These automatic captions are generally generated based on speech recognition technology.

[0003] Short video audio is often obtained by mixing user-generated audio with background music (BGM), and the two overlap significantly over a long period of time. Furthermore, the BGM may contain vocals or lyrics, resulting in the mixed signal containing both spoken and sung vocal components. Related research indicates that when musical accompaniment is present, tasks related to vocal / lyric recognition, as well as adjacent spoken language recognition, face greater challenges.

[0004] Existing multilingual speech recognition systems mostly employ end-to-end modeling, combined with large-scale pre-training or weakly supervised training to learn cross-language acoustic-text mapping capabilities. However, under the aforementioned conditions of spoken word and mixed audio containing human voice background music, the representations extracted by the model tend to carry both spoken language and lyrics information simultaneously. When the spoken word language and the lyrics language are different or there is code switching, cross-language lexical misinsertion, language drift, or semantic breaks are more likely to occur during the decoding stage, leading to a decrease in the usability of subtitles.

[0005] To address this type of interference, some existing solutions improve robustness by introducing mixed speech containing music into the training data, or by introducing music detection and speech enhancement / separation modules at the front end to suppress the accompaniment and vocal components. However, in short video mixes with distinct rhythms and strong coupling between accompaniment and vocals, front-end separation may introduce distortion or residual interference, and end-to-end models may still cause confusion at the boundaries of multilingual languages ​​and human voice types, thus limiting the effectiveness of multilingual subtitle generation. Summary of the Invention

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] This invention provides a method for constructing multilingual speech recognition models to solve the problems of miswritten lyrics and semantic breaks caused by mixing spoken audio with background music containing lyrics.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] This invention provides a method for constructing a multilingual speech recognition model, comprising:

[0010] Step S1: Obtain the mixed audio containing spoken human voice and background music, as well as the target language tag;

[0011] Step S2, constructing a speech recognition model, wherein constructing a speech recognition model includes: obtaining pre-trained parameters and loading them into a shared speech encoder, a first decoder and a second decoder, establishing the association between the first decoder and the first language model, and establishing the association between the second decoder and the second language model;

[0012] Step S3: The speech encoder extracts the acoustic feature sequence; the first decoder generates the first candidate text and the first time alignment information, and the second decoder generates the second candidate text and the second time alignment information.

[0013] Step S4: The first language model is used to perform probability evaluation on the two candidate texts within the same time window to obtain the first fluency score and the second fluency score. Based on the score difference and the threshold, a gating decision is made to determine the time window to be suppressed. The time window is a time interval divided along the mixed audio time axis according to a preset window length and a preset step size.

[0014] Step S5: Based on the first time alignment information, perform deletion or placeholder replacement processing on the first candidate text within the time window, and output the spoken language recognition text of the target language; wherein, the placeholder replacement processing is to replace the suppressed word segment with a placeholder word, and the placeholder word is a special word in the word table used to represent the suppressed non-spoken segment.

[0015] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, the mixed audio is divided into multiple overlapping time windows, and the first fluency score and the second fluency score are independently calculated for each time window, and the gating decision is made.

[0016] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, the first fluency score and the second fluency score are both scores obtained by taking the negative logarithmic measure of the conditional probability of the first language model for candidate text within a time window and normalizing it according to the number of lexical units. The smaller the score value, the more fluent the text is under the first language model.

[0017] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, the first time alignment information is generated by the frame-level timestamp or frame-level alignment result output by the first decoder during decoding, and is used to map the lexical units in the first candidate text to the time window.

[0018] In a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, the second language model is a lyrics language model, and the second decoder is used to generate the human voice lyrics text in the background music as the second candidate text.

[0019] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, wherein: when the second fluency score is lower than the first fluency score and the difference is greater than the threshold, the corresponding time window is determined to be the lyrics-dominated time window, and the suppression processing of the corresponding word element in the first candidate text is triggered;

[0020] In addition to using the difference between the two fluency scores and the threshold to make gating decisions during the gating decision stage, the difference is also mapped to a continuous gating strength. This continuous gating strength is then used to softly suppress the vocabulary output of the first decoder within the corresponding time window during the decoding process of generating the first candidate text. This causes the output probability of placeholder words to increase with the gating strength, while the output probability of non-placeholder words to decrease with the gating strength. The gating decision also introduces hysteresis boundaries or continuous time window consistent triggering constraints.

[0021] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, it further includes a dynamic context cache: when an external confirmation signal is received, the confirmed word, its language label and its corresponding acoustic feature sub-fragments are written into the cache.

[0022] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, wherein: the acoustic feature sub-segment vector is represented as a vector representation obtained by pooling, projection and normalization of frame-level acoustic features; during subsequent decoding, the cosine similarity between the current acoustic feature sub-segment and the acoustic feature sub-segments in the cache is calculated, and when the activation condition is met, the vocabulary probability distribution output by the first decoder in the current step is directionally biased;

[0023] The acoustic feature sub-fragment vector representation is obtained by pooling, projecting, and normalizing frame-level acoustic features, and is matched with the acoustic feature sub-fragment vector representation in the cache using cosine similarity; the activation condition is jointly limited by a similarity threshold and a continuous triggering constraint; when the directional bias is triggered, the bias strength has a monotonically corresponding relationship with the similarity, and is applied to the vocabulary output of the first decoder according to a decay law within several consecutive decoding steps; in the case of multilingual mixed decoding, the directional bias and the prior weight adjustment of the language seed vocabulary are superimposed in parallel, so that the probability of the sub-vocabulary of the activated language and the confirmed vocabulary simultaneously obtaining the same direction is increased.

[0024] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, the first decoder supports multilingual mixed decoding and divides the vocabulary into multiple language seed vocabulary lists; when the language tag of a cached entry is activated, the probability weight of the corresponding language seed vocabulary list in the current decoding step is adjusted.

[0025] As a preferred embodiment of the method for constructing a multilingual speech recognition model according to the present invention, wherein: when the target language is a low-resource language, a set of probabilistic sound change rules between the target language and related languages ​​is obtained; the initial candidate text generated by the first decoder for the mixed audio and represented in the related language is mapped by the sound change rules to obtain several constrained candidate texts, and the acoustic feature sequence is combined with the output of the first decoder for fusion scoring to determine the final spoken language recognition text;

[0026] The low-resource language is a language whose training corpus size is lower than a preset size threshold; the related language is a language that has a genealogical relationship with the low-resource language; the probabilistic sound change rule set is a set of sound change rules with occurrence probability weights; the constrained candidate text is a target language candidate text obtained by mapping the probabilistic sound change rule set; the fusion score is a comprehensive score obtained by combining acoustic consistency, first language model consistency and rule consistency.

[0027] Through the above technical solution, the present invention can achieve at least the following beneficial effects:

[0028] To address the issue that the spectral density of lyrics and spoken language is similar, and that end-to-end recognition can easily lead to the misinsertion of lyrics fragments into spoken language subtitles, this invention generates candidate text in parallel through spoken language branches and lyrics branches. Under the same spoken language model, the fluency of candidates within the window is evaluated comparatively. Gating is used to mark the time window that is more consistent with the lyrics generation rules as the suppression interval, thereby avoiding the contamination of spoken language subtitles by the semantics of lyrics at the output end.

[0029] To address the issues of unstable boundaries caused by rhythm and melody in mixed short videos, and the tendency of simple hard deletion to lead to subtitle breaks and segment jumps, this invention maps the gating difference to a continuous gating strength and applies this strength to the vocabulary output during the decoding process to form soft suppression. This makes non-verbal segments more likely to be output as placeholders rather than random verbal words. At the same time, it introduces hysteresis or continuous time window consistent triggering constraints to reduce gating jitter at the boundaries of overlapping time windows and improve subtitle coherence.

[0030] To address the issues in short video subtitle production, such as human-computer collaborative proofreading, repeated errors in colloquialisms / proper terms, and instability in cross-language word competition caused by code-switching, this invention utilizes dynamic context caching to store confirmed words, language tags, and corresponding acoustic segments. In subsequent decoding, it triggers directional bias and attenuation mechanisms based on acoustic matching, enabling confirmed words to be stably reused under similar acoustic conditions. This is combined with the weight adjustment of the language seed word list to improve language consistency and terminology consistency in code-switching scenarios.

[0031] To address the challenges of training stable recognition models directly for languages ​​with extremely low resource availability, and the vulnerability of multilingual shared modeling to data scarcity, this invention introduces related language intermediaries and probabilistic sound change constraints in low-resource scenarios. It generates target language constraint candidates through a restricted search and combines acoustic consistency, language model consistency, and rule consistency for fusion scoring, forming an executable constraint decoding path. This allows for the coverage of usable outputs for low-resource languages ​​without altering the robust mixing framework. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0033] Figure 1 This is a flowchart of the method for constructing a multilingual speech recognition model in the embodiments. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] Example 1:

[0037] like Figure 1 As shown, this application proposes a method for constructing a multilingual speech recognition model, including the following steps:

[0038] Step S1: Obtain the mixed audio containing spoken human voice and background music, as well as the target language tag;

[0039] The mixed audio is obtained from single-channel or multi-channel audio input and is processed by resampling and amplitude normalization to form an input waveform with a uniform sampling rate; the target language label is a single language label or a multi-language set label, which includes a language identifier. The language identifier is used to constrain the output space of the first decoder and to label the language of the output text.

[0040] In this embodiment, multi-channel audio is synthesized into a single-channel waveform before entering the speech encoder. The synthesis method involves weighting and summing the short-time energy of each channel and then normalizing the amplitude. The length of the energy statistics window for the weighted summation is an implementation parameter, defaulting to 0.4 seconds and adjustable within the range of 0.1 to 1.0 seconds. The target sampling rate for resampling is an implementation parameter, defaulting to 16000 Hz and adjustable within the range of 8000 to 48000 Hz. Amplitude normalization is achieved through peak value normalization, with a peak value target of 0.99 as the implementation parameter, adjustable within the range of 0.90 to 1.00. When the input audio has clipping bands, the peak value target is fixed at 0.95 and kept no higher than 1.00 to reduce distortion propagation. The time base for the mixed audio is set to 0 seconds from the audio start point, and all subsequent time range markers, time window markers, and frame indices are aligned with this time base.

[0041] Specifically, the target language tag is written by the upper-layer business system along with the audio to be recognized, or selected by the client user. The tag is written as a set of language identifiers. When the target language tag is a multilingual set, the language identifiers within the set are ordered according to their priority. This priority is used to determine the language seed word list that is preferentially opened in the first decoder's output space constraints. If the target language tag is missing or the set is empty, the target language tag is set as a default tag and is only used for language annotation in the output text.

[0042] Step S2, constructing a speech recognition model, which includes: obtaining pre-trained parameters and loading them into a shared speech encoder, a first decoder and a second decoder, establishing the association between the first decoder and the first language model, and establishing the association between the second decoder and the second language model;

[0043] The first decoder and the second decoder share a unified lexicalization rule and a unified lexical table. The lexical table contains multilingual seed lexicals, language tag lexicals, and placeholder lexicals. Placeholder lexicals are used to represent suppressed non-verbal segments. The first language model is a multilingual language model trained for spoken language transcription, and the second language model is a multilingual language model trained for domain-specific texts. Both the first decoder and the second decoder use the lexical table as their output space and output word-by-word confidence.

[0044] Furthermore, the unified lexicalization rule is implemented using sub-word segmentation rules. These sub-word segmentation rules and the lexical table are loaded as read-only configurations during system initialization. The size of the lexical table is an implementation parameter, defaulting to 64,000 and adjustable within the range of 8,000 to 200,000. The size is determined by the optimal balance between perplexity stability and decoding speed for the spoken language and lyrics branches on the development set. Language tag lexical units are a dedicated set of lexical units within the lexical table, with each language identifier corresponding one-to-one with a unique language tag lexical unit. Silence and blank lexical units are also dedicated lexical units within the lexical table. Silence lexical units represent speech pauses, blank lexical units represent empty outputs during decoding alignment, and placeholder lexical units represent suppressed non-spoken segments. Both the first and second language models employ the same lexicalization rules and lexical tables during training and inference, ensuring that the probability evaluation of two candidate texts under the first language model has the same lexical space reference basis.

[0045] For example, placeholder words in the output text take the form of a fixed string. The mapping between this fixed string and the placeholder word in the lexicon is determined during system initialization and remains unchanged. When placeholder words appear consecutively, they are merged into one placeholder word during post-processing to ensure consistent readability. Placeholder words do not participate in the conditional probability evaluation of the first language model and are not included in the number of content words, thus avoiding drift in the numerical scale of fluency scores caused by placeholder words.

[0046] Step S3: The speech encoder extracts the acoustic feature sequence. The acoustic feature sequence is obtained by the speech encoder from the input waveform frame by frame, and includes the hidden representation vector of each frame. The acoustic feature sequence maintains a frame-level correspondence with the input time axis and provides an index for subsequent time alignment, time window division, buffer matching, and suppression localization. The first decoder generates the first candidate text and the first time alignment information, and the second decoder generates the second candidate text and the second time alignment information. The time alignment information includes at least the word-level start frame index and end frame index, and the start frame index and end frame index correspond one-to-one with the frame index of the acoustic feature sequence. When the decoder uses CTC output, the time alignment information is generated by optimal path alignment. When the decoder uses attention or transduction output, the time alignment information is generated by the alignment trajectory under monotonic alignment constraints. When the alignment trajectory is missing, the time alignment information is generated by a forced alignment process based on word emission probability.

[0047] Similarly, frame division uses a fixed frame shift and a fixed frame length. The frame shift is an implementation parameter, defaulting to 0.01 seconds and adjustable within the range of 0.005 to 0.02 seconds; the frame length is an implementation parameter, defaulting to 0.025 seconds and adjustable within the range of 0.02 to 0.04 seconds. The conversion between frame index and time axis uses the frame shift as the time scale. The time position corresponding to any frame index is determined by the frame index and frame shift, and uses the same time base as the time range identifier. If the input waveform length is less than one frame, it is extended to one frame length by padding with zeros at the end, and a zero-padding mark is recorded. The zero-padding mark is only used for boundary alignment and does not participate in candidate text generation.

[0048] In this embodiment, the time unit corresponding to the start frame index and the end frame index is seconds, and it is obtained by converting frame shift and frame index. When the start frame index is less than 0 or the end frame index exceeds the maximum frame index of the acoustic feature sequence, the alignment result is clipped to the effective frame index range and a clipping mark is recorded. If the end frame index of a certain word is earlier than the start frame index, the alignment result of that word is invalidated and skipped in subsequent word-to-time window mapping. If multiple word alignment intervals overlap within the same time range, the overlapping area is allocated to the corresponding word according to the word confidence from high to low. The word confidence comes from the word-by-word confidence output by the decoder.

[0049] Step S4: The first language model is used to perform probability evaluation on the two candidate texts within the same time window to obtain the first fluency score and the second fluency score. Based on the score difference and the threshold, a gating decision is made to determine the time window to be suppressed. The time window is a time interval divided along the mixed audio time axis according to a preset window length and a preset step size. The probability evaluation evaluates the word sequence within the time window under a unified word segmentation rule. Before the evaluation, the candidate text is standardized, including uppercase and lowercase letter standardization, number standardization, whitespace standardization, and punctuation standardization. During the evaluation, silent words, whitespace words, and language tag words are ignored, and only content words are counted. The score is obtained by logarithmically measuring the conditional probability of the candidate sequence under the first language model and normalizing it according to the number of content words. The gating decision output includes a time window identifier and a suppression flag. The suppression flag indicates whether the first candidate text segment corresponding to the time window should be suppressed.

[0050] Furthermore, the standardization specification adopts unified number writing rules for numbers in the text, unifies full-width and half-width characters, compresses consecutive whitespace characters into a single whitespace character, and merges repeated punctuation marks. The punctuation set is an implementation parameter, which by default includes periods, commas, question marks, exclamation marks, and semicolons, and can be adjusted between retaining only periods and commas and retaining the entire set of commonly used punctuation marks. The standardization specification uses the same rules on both candidate texts to ensure the numerical comparability of the first and second fluency scores. When the candidate text contains language tag words, these language tag words are retained after standardization but ignored during probability evaluation and content word counting stages.

[0051] Specifically, the time window identifier is uniquely determined by the start and end times of the time window, and the time windows are numbered incrementally along the timeline for consistent referencing with subsequent alignment mappings. The window length and step size are implementation parameters. The default window length is 2.0 seconds, adjustable from 0.5 to 5.0 seconds, and the default step size is 0.5 seconds, adjustable from 0.1 to 1.0 seconds. The values ​​of the window length and step size are based on the quantile statistics of the shortest duration of lyric segments and the shortest stable duration of spoken segments on the development set. If the number of content tokens within a time window is 0, the suppression flag for that time window is fixed at a non-triggered state, and the state of the previous time window remains unchanged during the state machine smoothing phase.

[0052] Step S5: Based on the first time alignment information, perform deletion or placeholder replacement processing on the first candidate text within the time window, and output the spoken language recognition text in the target language; wherein, the placeholder replacement processing is to replace the suppressed word segment with a placeholder word, and the placeholder word is a special word in the word table used to represent the suppressed non-spoken segment.

[0053] When the suppression flag is in the triggered state, the content words in the first candidate text that fall within the time window are deleted or replaced by placeholder words; when placeholder words are replaced, consecutive placeholder words are merged into a single placeholder word, and the whitespace on both sides of the placeholder word is normalized to a single whitespace character; when deletion causes adjacent segments to be spliced, the punctuation and whitespace characters in the output text are reorganized by post-processing rules, which include removing duplicate punctuation, removing consecutive whitespace characters, and separating placeholder words from adjacent text.

[0054] In this embodiment, the mixed audio is divided into multiple overlapping time windows, and a first fluency score and a second fluency score are independently calculated for each time window, and a gating decision is made.

[0055] The time window is generated by a fixed window length and a fixed step size, and there is overlap between time windows; the gating decision adopts state machine smoothing, the state machine includes verbally dominant state and domain-dominant state, and has entry threshold and exit threshold; when multiple consecutive adjacent time windows meet the entry threshold trigger condition, the state enters the domain-dominant state, and when multiple consecutive adjacent time windows meet the exit threshold trigger condition, the state exits the domain-dominant state; the time window in the domain-dominant state is marked as a time window that needs to be suppressed.

[0056] In this embodiment, the fluency score is obtained by taking a negative logarithmic measure of the conditional probability of the candidate text within the time window by the first language model and normalizing it according to the number of lexical units. The smaller the score value, the more fluent the text is under the first language model.

[0057] The number of lexical units is the number of content lexical units within the time window; when the number of content lexical units within the time window is zero, the fluency score is set to the default value; when the candidate text contains placeholder lexical units, the placeholder lexical units are not included in the number of content lexical units and do not participate in the conditional probability evaluation; when the candidate text contains code mixing, the score is evaluated on the overall cross-language seed lexical sequence under the unified lexical table.

[0058] In this embodiment, the first time alignment information is generated by the frame-level timestamp or frame-level alignment result output by the first decoder during decoding, and is used to map the word units in the first candidate text to the time window;

[0059] The mapping process involves overlapping the start and end frame indices of each word with the frame range of the time window. When the overlap length reaches the preset overlap condition, the word is considered to belong to that time window. When a word spans multiple time windows, it is mapped to the time window with the longest overlap. When the overlap lengths are the same and parallel, the word is mapped to the earliest starting time window.

[0060] In this embodiment, the second language model is a lyrics language model, and the second decoder is used to generate the human voice lyrics text in the background music as the second candidate text;

[0061] The lyrics language model is trained on a corpus of lyrics texts, which includes lyrics texts in multiple languages ​​and structural tags. The second candidate text output by the second decoder contains lyrics content words and language tag words, and the second candidate text follows the same normalization and lexicalization rules as the first candidate text. The time alignment information of the second candidate text is output synchronously by the second decoder and is used to locate the lyrics candidate fragments to the time window.

[0062] In this embodiment, when the second fluency score is lower than the first fluency score and the difference is greater than the threshold, the corresponding time window is determined to be the lyrics-dominated time window, and suppression processing of the corresponding word element in the first candidate text is triggered.

[0063] The difference is obtained by subtracting the first fluency score from the second fluency score in a preset direction, which is fixed during system initialization. When the time window is determined to be a lyrics-dominated time window, the suppression process deletes or replaces the content words mapped to that time window. When the time window is determined to be a non-lyrics-dominated time window, the content words remain unchanged and enter the output post-processing.

[0064] In the gating decision stage, in addition to using the difference between the two fluency scores and the threshold to make gating decisions, the difference is also mapped to continuous gating strength. The continuous gating strength is used to softly suppress the vocabulary output of the first decoder in the corresponding time window during the decoding process of generating the first candidate text. This makes the output probability of placeholder words increase with the increase of gating strength, and the output probability of non-placeholder words decrease with the increase of gating strength. The gating decision also introduces hysteresis boundary or continuous time window consistent triggering constraint to reduce gating jitter at the boundary of overlapping time windows.

[0065] In one implementation, the gating decision is defined as follows: a lower score indicates greater fluency. This means the second fluency score is lower than the first fluency score, and the difference is greater than a threshold. This aligns the direction of the lyrics with the sign of the difference, avoiding semantic conflicts where a lower score indicates worse fluency within the same paragraph. Specifically:

[0066] When the time window index is At that time, the first language model is used to measure the average negative log probability of the word units within the time window of the candidate text, resulting in two fluency scores:

[0067] (1)

[0068] In equation (1), Indicates the first The fluency score of the first candidate text within each time window; the smaller the value, the more fluent the text is under the first language model. Indicates the first Fluency score of the second candidate text within a time window; Indicates the first The number of lexical units in the first candidate text within each time window; Indicates the first The number of lexical units in the second candidate text within each time window; Indicates the first The first candidate text within the first time window Each word element; This indicates the sequence of lexical terms preceding this lexical term; Indicates the first The second candidate text within the time window Each word element; This indicates the sequence of lexical terms preceding this lexical term; This represents the conditional probability given by the first language model; It represents the natural logarithm.

[0069] Under this definition, a lower second fluency score than the first fluency score indicates that the second candidate text has a lower average negative log probability under the first language model. Considering that the second candidate text originates from the lyrics decoding branch, the dominance of lyrics can be interpreted as follows: the acoustic content within this time window better conforms to the generation rules of the lyrics branch, and the word units output by the first decoder in this window are more prone to fragmentation, and spoken segments are more likely to be disturbed by musical beats, thus… Relative increase.

[0070] Based on this, the fixed difference direction is the subtraction of the second fluency score from the first fluency score, and the lyrics are determined to be dominant with a positive threshold.

[0071] (2)

[0072] In equation (2), Indicates the first The score difference over a time window, The larger the value, the more fluent the second candidate text is; Indicates the first The gating results of a time window This indicates the dominant time window of the lyrics. Indicates a time window where lyrics do not dominate; This indicates an indicator function that returns 1 if the condition within the parentheses is true, and 0 otherwise. This represents the gating threshold, used to limit the minimum difference required to trigger lyric dominance.

[0073] To reduce boundary jitter caused by overlapping time windows, hysteresis gating can be introduced without changing the decision structure, replacing the single threshold with a trigger threshold / release threshold:

[0074] (3)

[0075] In equation (3), Indicates the first Gating results for each time window; This indicates the trigger threshold for switching from non-lyrics-dominated to lyrics-dominated mode. This indicates the release threshold for switching from lyrics-driven to non-lyrics-driven, satisfying... It can suppress boundary back-and-forth switching.

[0076] Continuous can also be used Individual window triggers alternative hysteresis, using count consistency to filter isolated noise windows:

[0077] (3.1)

[0078] In equation (3.1), Indicates the first Each time window is at the end and has a length of [missing information]. The number of windows in the window sequence that are determined to be dominated by lyrics; This represents the length of the continuous decision window, which is a positive integer. This represents the gating result after continuous window constraints. This represents the gating result after continuous window constraint.

[0079] When it is necessary to extend the difference greater than a threshold into a probabilistic gating system for soft suppression, the difference can be mapped to a gating coefficient to control the suppression strength instead of a binary switch.

[0080] (4)

[0081] In equation (4), Indicates the first The gating coefficients for each time window, with values ​​ranging from [value range missing]. The larger the value, the higher the degree of influence of the lyrics; This represents the sigmoid slope coefficient, used to adjust the sensitivity of the difference change to the gating strength; Indicates the sigmoid center threshold; Represents the natural constant.

[0082] When gating coefficients are used for soft suppression of the first decoder, placeholder words can be targeted for soft suppression, compatible with the deletion or placeholder replacement steps described above. This makes the output more likely to produce placeholder words rather than spoken words in the lyrics-dominant window. Align to the first The set of frame indices for each time window is denoted as . The first decoder in the frame The vocabulary logit vector is The word index of placeholder words is Logit can then be adjusted according to the following rules:

[0083] (5)

[0084] In equation (5), Representing a frame The above term index is The original logit; This represents the adjusted logit; Representing a frame The logit vector; This indicates that the information is obtained by mapping the first-time alignment information and falls into the first... A set of frame indices for each time window; This indicates the index of the placeholder word in the first decoder vocabulary; Indicates the space enhancement factor; Indicates the non-occupancy suppression coefficient; The aforementioned gating coefficients will be used. This adjustment will increase the competitiveness of placeholder words within the lyrics-dominant window. By increasing the logit gain and combining it with deletion or placeholder replacement processing, the deletion processing can be changed to increase the posterior probability of placeholder words through logit gain, thereby reducing text jumps at the window boundary.

[0085] Within each time window, based on the two candidate texts output by the first decoder and the second decoder and their time alignment information, two fluency scores are obtained according to Equation (1), and a common difference is formed according to Equation (2) to determine the degree of lyric dominance. When jitter suppression is used, the gating result is introduced with hysteresis or continuous triggering constraints according to Equation (3) or Equation (3.1) to avoid repeated switching of adjacent overlapping windows at the boundary. When it is necessary to transition from binary gating to continuous gating, the difference is mapped to a gating coefficient according to Equation (4) in the gating decision stage. This gating coefficient is applied to the vocabulary output of the first decoder according to Equation (5) in the decoding output modulation stage, so that the posterior probability of the target occupier word in the lyric dominance time window increases and the posterior probability of the non-occupier word decreases. Then, the suppression range is limited to the corresponding time window according to the first time alignment information, and the spoken language recognition text of the target language is output in the deletion or occupier replacement process. The gating threshold and hysteresis boundary are preset by the method configuration stage, and the number of continuous triggering windows is preset by the method configuration stage.

[0086] Specifically, in the above implementation, the gating judgment compares the fluency of two candidate texts under the same language model in units of time windows, and defines the score as the smaller the score, the more fluent it is. Thus, if the second score is lower than the first score and the difference exceeds the threshold, the corresponding lyrics branch naturally corresponds to a situation where it is more consistent with the acoustic content of that window. After fixing the direction of the difference as the first minus the second, an increase in the positive difference value indicates that the lyrics dominance is enhanced, and the semantic judgment remains coherent. To reduce the back-and-forth switching of overlapping time windows at the boundary, trigger and release thresholds can be introduced to form hysteresis, or constraints of consistent triggering of multiple consecutive time windows can be used to suppress isolated misjudgments. On this basis, the difference can also be mapped to continuous gating strength to perform soft suppression on the output of the first decoder, so that the lyrics dominance window tends to output placeholders rather than spoken words, reducing the sense of discontinuity caused by deletion processing, and maintaining consistency with the in-window processing based on time alignment information.

[0087] In this embodiment, a dynamic context cache is also included: when an external confirmation signal is received, the confirmed word, its language tag and its corresponding acoustic feature sub-fragments are written into the cache;

[0088] The external confirmation signal includes the text of the confirmed word, the language tag of the confirmed word, and the time range identifier corresponding to the confirmed word; the time range identifier is generated by the subtitle editing confirmation event, the review confirmation event, or the retrieval confirmation event and carries an audio timestamp; the acoustic feature sub-segment is obtained by mapping the time range identifier to the acoustic feature sequence frame index; each entry in the context cache includes an entry identifier, a word lexical sequence, a language tag, an acoustic feature sub-segment vector representation, and an entry confidence flag; when the cache capacity reaches the preset limit, eviction is performed according to the entry confidence flag and the order of most recently used.

[0089] In this embodiment, the text of the confirmed word comes from the text field of the subtitle editing confirmation event, review confirmation event, or retrieval confirmation event, and undergoes the same normalization processing as the consistency specification before being written to the context cache. The time range identifier is a time period consisting of a start time and an end time, with the time unit being seconds. The time precision is 0.01 seconds by default and can be adjusted within the range of 0.001 to 0.1 seconds. This time precision is an implementation parameter and is determined according to the timestamp precision of the upper-layer event system. When the time range identifier is mapped to the acoustic feature sequence frame index, frame shift conversion is used and the boundaries are clipped. When the end time is earlier than the start time or the time period length is 0, the external confirmation signal is determined to be invalid and is not written to the cache. If the same confirmed word is written to the cache multiple times under the same language tag, the latest external confirmation signal overwrites the time range identifier of the old entry and refreshes the most recently used order.

[0090] In this embodiment, the acoustic feature sub-segment vector is represented by a vector representation obtained by pooling, projection and normalization of frame-level acoustic features; during subsequent decoding, the cosine similarity between the current acoustic feature sub-segment and the acoustic feature sub-segments in the cache is calculated. When the activation condition is met, the word probability distribution output by the first decoder in the current step is directionally biased to increase the output probability of the confirmed word.

[0091] The acoustic feature sub-segment vector representation is obtained by pooling the acoustic feature sequence within the sub-segment frame and performing vector normalization; similarity calculation calculates the cosine similarity between the current sub-segment and the vector representation of each entry in the cache, and selects the entry with the highest similarity; the activation condition includes the highest similarity exceeding a preset similarity threshold and the difference between the highest similarity and the second highest similarity exceeding a preset interval threshold; when the activation condition is met, the orientation bias improves the probability quality of the confirmed word unit corresponding to the current step of the first decoder in the output vocabulary, and renormalizes the probability quality of the remaining word units; the orientation bias remains effective within a preset duration and decays as the decoding step progresses until it becomes invalid.

[0092] The acoustic feature sub-fragment vector representation is obtained from frame-level acoustic features through pooling, projection, and normalization, and is matched with the acoustic feature sub-fragment vector representation in the cache using cosine similarity; the activation condition is jointly limited by the similarity threshold and the continuous triggering constraint; when the directional bias is triggered, the bias strength has a monotonically corresponding relationship with the similarity, and is applied to the vocabulary output of the first decoder according to the decay law in several consecutive decoding steps to increase the output probability of the corresponding word of the confirmed word; in the case of multilingual mixed decoding, the directional bias and the prior weight adjustment of the language seed vocabulary are superimposed in parallel, so that the probability of the sub-vocabulary of the activated language and the confirmed word simultaneously obtaining the same direction is increased;

[0093] In one implementation, the directional biasing process allows cache entries to be organized according to the identified vocabulary, language tags, and acoustic feature sub-fragment vectors. As follows:

[0094] The cache entry index is At that time, the cached content is recorded as ,in For confirmed words, For the language tag of this word, This is the vector representation of the acoustic feature sub-fragment corresponding to the word. The index of the currently decoded associated acoustic sub-fragment is... The frame-level acoustic features output by the acoustic encoder are The set of frame indices corresponding to the sub-fragments is The acoustic feature sub-fragment vector representation can be obtained by pooling, projection, and normalization:

[0095] (6)

[0096] In equation (6), Sub-fragment Pooling characteristics; Represents a set The number of elements; Represents the acoustic projection matrix; Represents the acoustic projection bias vector; Represents the projected sub-fragment vector; This represents a sub-segment vector after being normalized to L2 norm; This represents the L2 norm.

[0097] In obtaining Then, compare it with the cache vector. Calculate the cosine similarity and select the most similar entries:

[0098] (7)

[0099] In equation (7), Sub-fragment With cache entries Cosine similarity; Indicates transpose; This represents the index of the cache entry most similar to the current sub-fragment; This represents the maximum similarity.

[0100] The activation condition uses a combination of a similarity threshold and continuous triggering. Let the single trigger indicator be... And use continuous counting to suppress isolated triggering:

[0101] (8)

[0102] In equation (8), Sub-fragment Has the similarity threshold been reached? Indicates an indicator function; Indicates the similarity threshold; Indicates the end of the sub-fragment The continuous trigger count; This indicates the consecutive trigger count of the previous sub-segment; Indicates the consecutive trigger threshold, which is a positive integer; Indicates the activation result. This indicates that a directional bias is triggered. This indicates that it will not be triggered.

[0103] When the directional bias falls into a computable operation, the vocabulary will be identified. The set of lexical indices corresponding to the first decoder vocabulary is denoted as .

[0104] The first decoder in the decoding step The vocabulary logit is denoted as ,in This is a vocabulary index. A monotonic mapping is used from similarity to bias strength, so that higher similarity results in a larger bias.

[0105] ,

[0106] A decay is applied over the duration of the bias, where the duration of the bias is 1. The attenuation factor is For the first step from the current step The subsequent decoding steps use Attenuation, then:

[0107] (9)

[0108] In equation (9), This represents the magnitude of the lexical item bias obtained from the similarity mapping; Indicates the maximum bias coefficient of the lexical item; Indicates the duration of the bias step; This represents the bias attenuation factor, with a value range of [value range missing]. ; This represents the offset relative to the current decoding step; This represents the logit after applying bias. The adjusted logit, after being processed by the softmax normalization function, represents the following: The probability of internal lexical elements increases, thereby improving... The output probability.

[0109] In multilingual scenarios, the weighting of the language seed vocabulary can be adjusted in parallel with the aforementioned vocabulary item biasing. The language is labeled as... The set of sub-word indexes is denoted as When the language of the activated entry is At that time, a language prior bias is introduced for this sub-vocabulary:

[0110] (10)

[0111] In equation (10), The language tag is The set of sub-vocabulary indexes; Indicates the language bias magnitude; Indicates the language scaling factor; This represents the logit after the language item is biased. This represents the index of the cache entry most similar to the current sub-fragment; This indicates the language tag corresponding to the most similar cached entry; Indication and language label The corresponding language seed vocabulary index set. This joint bias simultaneously enhances the competitiveness of the target vocabulary's corresponding lexical unit and the target language seed vocabulary in the current decoding step when the language tag is activated, ensuring that the bias direction is consistent with the language tag carried by the cached entry.

[0112] If you need to replace adding a constant to logit with interpolation of the probability distribution, in The same can be used at the same time First, normalize it into interpolation coefficients, then transform the original probability distribution to... Centralized offset; this approach corresponds to a soft boost in effect, similar to logit offset. and Under constraints, the bias remains consistent with the decay logic.

[0113] As can be seen, when an external confirmation signal is received, the confirmed words, language tags and their corresponding acoustic sub-segments are saved in the dynamic context cache writing stage. In subsequent decoding, for the current acoustic sub-segment, the sub-segment vector is constructed from the frame-level acoustic features according to Equation (6) and normalized. Then, the cosine similarity with the cache entries is calculated according to Equation (7) and the most similar entry is selected. In the similarity matching stage, the activation determination adopts the similarity threshold according to Equation (8) and superimposes the continuous trigger constraint so that the occasional similarity peak does not directly trigger the bias. When the activation condition is met, the bias intensity is obtained by the similarity mapping according to Equation (9) in the bias generation stage, and is continuously applied to the word list output of the first decoder according to the decay law in several decoding steps, so that the posterior probability of the target word corresponding to the word increases with the increase of similarity. In the multilingual mixed decoding scenario, the language prior bias and the word item bias are superimposed in the same decoding step. According to Equation (10), the prior gain in the same direction is introduced to the sub-word list of the activated language so that the word item improvement and the language sub-word list improvement appear synergistically. The similarity threshold, bias duration steps, and decay factor are preset in the method configuration phase. The writing of cache entries is triggered by the dynamic context cache writing process when an external acknowledgment signal is received. The bias is applied during the subsequent decoding process.

[0114] Specifically, the bias mechanism in the above implementation constructs comparable vector representations on an acoustic sub-segment basis. These vectors are obtained by linearly projecting and normalizing the pooled acoustic features, ensuring that cosine similarity reflects the acoustic proximity between the current segment and the cached segments. The activation condition employs both a similarity threshold and a continuous trigger count; occasional high similarity at window or segment boundaries does not directly trigger the bias, thus reducing output jitter. The directional bias falls on a computable operation on the output of the first decoder's vocabulary. By applying a positive gain to the logit of the corresponding lexical unit of the target word, it achieves a higher output probability after normalization. The duration and intensity of the bias are controlled by the number of steps and a decay factor. In multilingual hybrid decoding, the lexical item bias and the language seed vocabulary prior bias are superimposed, allowing the language and lexical information of the activated entries to work together in the same decoding process, improving the reuse stability and consistency of cached entries in subsequent decoding.

[0115] In this embodiment, the first decoder supports multilingual mixed decoding and divides the vocabulary into multiple language seed vocabulary lists; when the language tag of a cached entry is activated, the probability weight of the corresponding language seed vocabulary list in the current decoding step is adjusted.

[0116] The language seed vocabulary is a subset of the unified vocabulary table divided by language labels; the language weight adjustment increases the probability weight of the sub-vocabulary corresponding to the activated language and decreases the probability weight of the sub-vocabulary corresponding to the inactive language, and normalizes the weights of all sub-vocabularies; when the language posterior output by the first decoder is inconsistent with the activated language label, the language weight adjustment is dominated by the activated language label and updates the language posterior under the preset conflict handling rules.

[0117] Example 2:

[0118] Based on Example 1, when the target language is a low-resource language, a set of probabilistic sound change rules between the target language and related languages ​​is obtained; the initial candidate text generated by the first decoder for the mixed audio and represented in the related language is mapped by sound change rules to obtain several constrained candidate texts, and the acoustic feature sequence is combined with the output of the first decoder for fusion scoring to determine the final spoken language recognition text.

[0119] Low-resource languages ​​are those whose training corpus size is below a preset size threshold; related languages ​​are those with a lineage relationship to low-resource languages; the probabilistic sound change rule set is a set of sound change rules with accompanying probability weights; constrained candidate texts are candidate texts of the target language obtained by mapping the probabilistic sound change rule set; the fusion score is a comprehensive score obtained by combining acoustic consistency, first language model consistency, and rule consistency.

[0120] When the target language is determined to be a low-resource language, the related languages ​​are determined by a preset genealogical table; the probabilistic sound change rule set is represented in the form of weighted rules and includes the context information of rule application; the first candidate text is first converted into a sequence of phonemes or sub-word units, and then a target language unit sequence is generated and decoded in the rule-constrained search space to obtain the constrained candidate text; the restricted search space uses pruning rules to limit candidate expansion and maintain a preset bundle width; the fusion score calculates the acoustic consistency score, the first language model score and the rule consistency score for the constrained candidate text respectively and performs a comprehensive ranking, and the constrained candidate text with the highest comprehensive ranking is determined as the final spoken language recognition text.

[0121] In this embodiment, the weights of the weighted rules range from 0.01 to 0.99. Weights are not set to 0 to prevent the rule from never taking effect, and are not set to 1 to prevent the constraint candidate space from degenerating due to complete rule determination. Rule weights are normalized to a sum of 1.00 under the same source unit to ensure the comparability of probabilities of replacing different target units. The applicable context information for the rules is used to limit the adjacent unit conditions for rule triggering; these adjacent units come from phoneme or sub-word unit sequences. When context information is missing, the rule is considered a context-independent rule and can be triggered at any position. The probabilistic sound change rule set is loaded as a read-only configuration during system initialization and associated with a preset genealogical relationship table by key-value index of the target language and related languages.

[0122] Furthermore, the preset bundle width is an implementation parameter, defaulting to 16 and adjustable within the range of 4 to 64. The bundle width value is based on the joint optimality of constraint candidate coverage and computational cost on the development set. The pruning rules retain only the top-scoring partial candidates within the first bundle width for each expansion step, and deduplicate duplicate partial candidate sequences, retaining only the highest-scoring entry. If the number of candidates after an expansion step is less than the bundle width, all candidates are retained, and the next expansion step continues. If the candidate set is empty due to rule constraints, the output of this mapping process is an empty set, triggering a backtrack. The backtrack directly uses the initial candidate text as constraint candidate text in subsequent fusion scoring.

[0123] For example, the acoustic consistency score is obtained by measuring the consistency between the constrained candidate text and the acoustic feature sequence on the decoder output likelihood; the first language model score is obtained by measuring the conditional probability of the constrained candidate text by the first language model; and the rule consistency score is obtained by accumulating the weights of the constrained candidate text on the sound change rule triggering path. All three are normalized in the same ranking direction before participating in the comprehensive ranking. The comprehensive ranking uses a weighted sum to obtain the comprehensive score. The fusion weights are implementation parameters, with a default value of 0.5 for acoustic consistency, 0.3 for the first language model, and 0.2 for rule consistency. Each weight can be adjusted within the range of 0.0 to 1.0, and the sum must be 1.0. The fusion weights are determined by the value on the development set that achieves a preset balance between the language consistency and readability of the final spoken language recognition text. If there is a tie for the best comprehensive ranking, the one with the better first language model score is given priority; if there is still a tie, the one with the better acoustic consistency score is given priority; if the constrained candidate text is an empty set, the final spoken language recognition text is the initial candidate text, and the comprehensive ranking is skipped.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0125] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for constructing a multilingual speech recognition model, characterized in that, include: Step S1: Obtain the mixed audio containing spoken human voice and background music, as well as the target language tag; Step S2, constructing a speech recognition model, wherein constructing a speech recognition model includes: obtaining pre-trained parameters and loading them into a shared speech encoder, a first decoder and a second decoder, establishing the association between the first decoder and the first language model, and establishing the association between the second decoder and the second language model; Step S3: The speech encoder extracts the acoustic feature sequence; the first decoder generates the first candidate text and the first time alignment information, and the second decoder generates the second candidate text and the second time alignment information. Step S4: The first language model is used to evaluate the probability of the two candidate texts within the same time window to obtain the first fluency score and the second fluency score. The gating decision is made based on the score difference and the threshold to determine the time window to be suppressed. The time window is a time interval divided along the mixed audio time axis according to a preset window length and a preset step size. Step S5: Based on the first time alignment information, perform deletion or placeholder replacement processing on the first candidate text within the time window, and output the spoken language recognition text of the target language; wherein, the placeholder replacement processing is to replace the suppressed word segment with a placeholder word, and the placeholder word is a special word in the word table used to represent the suppressed non-spoken segment; The mixed audio is divided into multiple overlapping time windows, and the first fluency score and the second fluency score are calculated independently for each time window, and the gating decision is made accordingly. Both the first fluency score and the second fluency score are obtained by taking a negative logarithmic measure of the conditional probability of the candidate text within the time window by the first language model and normalizing it according to the number of lexical units. The smaller the score value, the more fluent the text is under the first language model.

2. The method for constructing a multilingual speech recognition model according to claim 1, characterized in that, The first time alignment information is generated by the frame-level timestamp or frame-level alignment result output by the first decoder during decoding, and is used to map the lexical units in the first candidate text to the time window.

3. The method for constructing a multilingual speech recognition model according to claim 1, characterized in that, The second language model is a lyrics language model, and the second decoder is used to generate the vocal lyrics text in the background music as the second candidate text.

4. The method for constructing a multilingual speech recognition model according to claim 3, characterized in that, When the second fluency score is lower than the first fluency score and the difference is greater than the threshold, the corresponding time window is determined to be the lyrics-dominated time window, and suppression processing of the corresponding word element in the first candidate text is triggered. In the gating decision stage, the difference is mapped to a continuous gating strength, and the continuous gating strength is used to softly suppress the vocabulary output of the first decoder in the corresponding time window during the decoding process of generating the first candidate text, so that the output probability of placeholder words increases with the increase of gating strength, and the output probability of non-placeholder words decreases with the increase of gating strength; the gating decision also introduces hysteresis boundary or continuous time window consistent triggering constraint.

5. The method for constructing a multilingual speech recognition model according to claim 1, characterized in that, It also includes a dynamic context cache: when an external acknowledgment signal is received, the acknowledgment word, its language tag and its corresponding acoustic feature sub-fragments are written into the cache.

6. The method for constructing a multilingual speech recognition model according to claim 5, characterized in that, The acoustic feature sub-segment vector is represented by a vector obtained by pooling, projection and normalization of frame-level acoustic features; during subsequent decoding, the cosine similarity between the current acoustic feature sub-segment and the acoustic feature sub-segments in the cache is calculated, and when the activation condition is met, the vocabulary probability distribution output by the first decoder in the current step is directionally biased; The acoustic feature sub-fragment vector representation is obtained by pooling, projecting and normalizing the frame-level acoustic features, and is matched with the acoustic feature sub-fragment vector representation in the cache using cosine similarity; the activation condition is jointly limited by the similarity threshold and the continuous triggering constraint; when the directional bias is triggered, the bias intensity is monotonically correlated with the similarity, and is applied to the vocabulary output of the first decoder according to the decay law in several consecutive decoding steps; In the case of multilingual mixed decoding, the directional bias and the prior weight adjustment of the language seed word list are superimposed in parallel, which increases the probability that the sub-word list of the activated language and the confirmed words will simultaneously obtain the same direction.

7. A method for constructing a multilingual speech recognition model according to claim 5 or 6, characterized in that, The first decoder supports multilingual mixed decoding and divides the vocabulary into multiple language seed vocabularys; when the language tag of a cached entry is activated, the probability weight of the corresponding language seed vocabulary in the current decoding step is adjusted.

8. The method for constructing a multilingual speech recognition model according to claim 1, characterized in that, When the target language is a low-resource language, obtain the set of probabilistic sound change rules between the target language and related languages; map the initial candidate text generated by the first decoder for the mixed audio in the related language through the sound change rules to obtain several constrained candidate texts, and combine the acoustic feature sequence with the output of the first decoder to perform fusion scoring to determine the final spoken language recognition text; The low-resource language is a language whose training corpus size is lower than a preset size threshold; the related language is a language that has a genealogical relationship with the low-resource language; the probabilistic sound change rule set is a set of sound change rules with occurrence probability weights; the constrained candidate text is a target language candidate text obtained by mapping the probabilistic sound change rule set; the fusion score is a comprehensive score obtained by combining acoustic consistency, first language model consistency and rule consistency.

Images