Training Methods for Multilingual Speech Recognition Models Based on Posterior and Text Simulation
By adopting a multilingual speech recognition model training method that shares a posterior simulator and projector within the same connected temporal classification vocabulary space, the problems of scarce training resources and cross-language imbalance in multilingual speech recognition models are solved, and a unified interface for cross-language alignment and improved recognition performance are achieved.
Patent Information
- Application Number
- CN202611355170.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-10
AI Technical Summary
In existing technologies, training resources for multilingual speech recognition models are scarce and uneven across languages, resulting in high requirements for training data, high parameter count and maintenance costs. In addition, the dense storage and computational overhead of native multilingual CTC vocabularies are also significant.
A multilingual speech recognition model training method based on posterior and text simulation is adopted. By sharing the posterior simulator, projector and large language model in the same connected temporal classification vocabulary space, the modality switching optimization training is performed using the text generation simulation posterior in the multilingual training set. Then, the distribution mismatch optimization training is performed using the real posterior, and a cross-language shared projector and large language model are established.
It reduces the reliance on paired speech, improves multilingual recognition performance, reduces the burden of posterior access and projection, achieves a unified interface for cross-language alignment, and enhances training reusability and recognition results.
Smart Images

Figure CN123072372A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent speech, and in particular to a method, system, electronic device, and storage medium for training a multilingual speech recognition model based on posterior and text simulation. Background Technology
[0002] Speech LLMs (Large Language Models) extend large language models to speech processing tasks. By aligning speech representations to the input space of the LLM, they have achieved superior results in speech recognition and speech understanding. Common techniques used include: 1. Continuous acoustic feature projection in large speech language models: Speech input first passes through an acoustic encoder to form a high-frequency, long-sequence continuous hidden representation, then is mapped to the input space of the large language model through modules such as linear layers, and finally the large language model generates the transcription. This typically requires a large amount of speech-text pairing data to learn cross-modal mappings. In multilingual scenarios, the amount of pairing data varies significantly between languages.
[0003] 2. Utilizing the compact posterior representation of Connectionist Temporal Classification (CTC), the CTC encoder outputs a probability distribution containing blank characters and vocabulary symbols for each speech frame. By removing blank-dominant frames and merging consecutive repeating labels, the input sequence can be significantly shortened. The compressed CTC posterior is then projected into the embedding space of a large language model, thereby reducing the temporal redundancy of continuous acoustic features.
[0004] 3. TASU (Text-only Alignment for Speech Understanding) and TASU2. TASU segments text according to the CTC vocabulary and then constructs a simulated CTC posterior through posterior smoothing, random deletion, and random insertion to train the posterior projector; during inference, the simulated posterior is replaced with the real CTC posterior generated by the speech. TASU2 uses a learning simulator that predicts the posterior sequence from the text to learn duration, blank distribution, label confusion, and contextual dependencies. Existing validations, while including speech recognition, sentiment assessment, and speech understanding, mainly focus on English and have not formed a complete scheme for sharing alignment interfaces and performing a small amount of real posterior calibration within the same native CTC posterior space for multiple languages.
[0005] 4. Multilingual CTC Automatic Speech Recognition: Multilingual CTC models can handle multiple languages in a shared vocabulary and control the recognition task with language symbols and task symbols. Taking OWSM-CTC as an example, it is trained on 151 languages and approximately 180,000 hours of audio, and provides a native 50,002-class CTC posterior space.
[0006] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology: 1. Continuous feature projection directly uses the hidden state of the acoustic encoder as the alignment input. Low-resource languages have difficulty obtaining sufficient and high-quality data, which requires a large amount of training data and has a strong dependence on paired data. 2. When mixing multilingual paired speech, the high-resource language tends to dominate the optimization. However, building an independent projector for each language will increase the number of parameters, training costs, and maintenance costs with the number of languages, making it difficult to achieve an even distribution of languages. 3. The simulated posterior generated by text initialization is not completely consistent with the real audio posterior. Directly extending English TASU to multiple languages cannot guarantee effective shared alignment. In other words, there are distributional differences between the training and inference interfaces. 4. The native multilingual CTC vocabulary can contain tens of thousands of categories. Storing it offline in a dense format and repeatedly projecting it will increase the amount of data loading and matrix operations, resulting in a large overhead for speech posterior sequence calculation and storage.
[0007] In summary, mainstream alignment paradigms rely on continuous feature projection modules trained from audio-text pairing data. Paired audio-text resources are not only scarce, but also suffer from severe cross-language imbalance, with many languages lacking sufficient data to train dedicated alignment modules. Summary of the Invention
[0008] In order to at least address the scarcity of resources and the imbalance across languages in the training of multilingual speech recognition models in existing technologies.
[0009] In a first aspect, embodiments of the present invention provide a method for training a multilingual speech recognition model based on posterior and text simulation, comprising: The multilingual training set is input into the multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters across multiple languages, and a large language model; In the first training phase, multilingual simulated posteriors are generated using only the text in the multilingual training set through the posterior simulator. The projector is then trained using modality switching optimization through the multilingual simulated posteriors to obtain a projector with shared cross-language alignment. In the second training phase, the true posterior is determined by the posterior provider using a paired text-speech sample from another text-speech sample in the multilingual training set. The projector distribution mismatch is then optimized and trained using the true posterior to obtain the trained multilingual speech recognition model.
[0010] Secondly, embodiments of the present invention provide a multilingual speech recognition model training agent system based on posterior and text simulation, comprising: A speech recognition module is used to input multilingual training sets into a multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters for multiple languages, and a large language model; The training module is used in the first training phase to generate a multilingual simulated posterior using only text from the multilingual training set through the posterior simulator, and to perform modality switching optimization training on the projector using the multilingual simulated posterior to obtain a projector with shared cross-language alignment. In the second training phase, the module uses a text-speech pairing sample from another text in the multilingual training set to determine the true posterior using the posterior provider, and continues to optimize the projector distribution mismatch training using the true posterior to obtain the trained multilingual speech recognition model.
[0011] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multilingual speech recognition model training method based on posterior and text simulation according to any embodiment of the present invention.
[0012] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the multilingual speech recognition model training method based on posterior and text simulation according to any embodiment of the present invention.
[0013] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the multilingual speech recognition model training method based on posterior and text simulation according to any embodiment of the present invention.
[0014] The beneficial effects of this invention are as follows: This method establishes a unified cross-language interface, allowing multiple target languages to share projectors, large language models, and LoRA parameters within the same native CTC posterior space. This eliminates the need to set up independent alignment modules for each language, reducing dependence on paired speech and improving multilingual recognition performance. Furthermore, it improves the quality of simulated posteriors, reduces the burden of posterior access and projection, and enhances training reusability. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart of a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall multilingual TASU framework for a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the training speech sample size for each language in a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of a general model and posterior configuration for a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of label synchronous compression and projection of a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the text-first initialization and a small amount of real posterior calibration process of a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention. Figure 7 This is a schematic diagram of the multilingual TASU shared posterior interface and the overall process of training, calibration and inference of a multilingual speech recognition model training method based on posterior and text simulation provided by an embodiment of the present invention. Figure 8 This is a schematic diagram of the word error rate under the condition of freezing the Qwen backbone network and training only the posterior projector, which is a training method for a multilingual speech recognition model based on posterior and text simulation provided by an embodiment of the present invention. In this diagram, Sim. represents the multilingual simulator obtained after learning. Figure 9 This is a schematic diagram of the word error rate based on LoRA and TASU adaptation for a multilingual speech recognition model training method based on posterior and text simulation provided in an embodiment of the present invention. Figure 10 This is a schematic diagram illustrating the impact of removing Polish from simulator training in a multilingual speech recognition model training method based on posterior and text simulation provided by an embodiment of the present invention. Figure 11 This is a schematic diagram of the word error rate when using the Whisper-CTC front-end in a training method for a multilingual speech recognition model based on posterior and text simulation provided in an embodiment of the present invention. Figure 12 This is a schematic diagram of the structure of a multilingual speech recognition model training agent system based on posterior and text simulation, provided in an embodiment of the present invention. Figure 13 This is a schematic diagram of an embodiment of an electronic device trained using a multilingual speech recognition model based on posterior and text simulation, as provided in an embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] like Figure 1 The diagram shown is a flowchart of a multilingual speech recognition model training method based on posterior and text simulation according to an embodiment of the present invention, including the following steps: S11: Input the multilingual training set into the multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distribution is in the same connection temporal classification vocabulary space, a projector and a large language model that share the same parameters for multiple languages; S12: In the first training phase, a multilingual simulated posterior is generated using only the text in the multilingual training set through the posterior simulator. The projector is then trained using modality switching optimization through the multilingual simulated posterior to obtain a projector with shared cross-language alignment. S13: In the second training phase, the true posterior is determined by the posterior provider using another text-speech pairing sample in the multilingual training set. The projector distribution mismatch is then optimized and trained using the true posterior to obtain the trained multilingual speech recognition model.
[0019] This method addresses the scarcity of training resources and the imbalance across languages in existing speech-based large language models. It requires a solution that unifies text and speech inputs within the native multilingual CTC posterior space, shares a projector and large language model adaptation across languages, and allows for alignment establishment using multilingual text followed by calibration with a small amount of real posterior data. To achieve this, simply put, this method uses the native posterior space of the multilingual CTC encoder as a cross-language shared interface, enabling all target languages to share a single posterior projector, a single language model, and shared low-rank adaptation parameters. In the same posterior space, the first stage of training is performed using simulated posterior data generated from text, followed by a second stage of calibration using real posterior data generated from a small amount of speech. During inference, the real posterior data is always used, coupled with label-synchronized compression and Top-K sparse projection. These steps together form a complete technical path where training and inference inputs are interchangeable, parameters are shared across languages, and a large amount of text can be used for initialization. It should be noted that TASU has only been validated for English. Directly applying TASU to multilingual speech recognition—for example, simply applying existing text alignment methods to the native posterior space of multilingual CTC without other modifications—will result in even worse recognition performance. All steps in this method are designed to address the issue of TASU's ineffectiveness in multilingual languages.
[0020] Regarding step S11, considering the aforementioned technical features that this method focuses on, such as Figure 2 As shown, it illustrates the multilingual speech recognition model of this method, which can also be referred to as the multilingual TASU (Text-only Alignment for Speech Understanding) overall model architecture and various training strategies to be evaluated.
[0021] The multilingual speech recognition model of this method includes a posterior simulator and a posterior provider whose output posterior distributions are located in the same CTC vocabulary space. The projector and large language model share a set of parameters for all target languages. There are no language-specific projectors or output heads. In other words, this method prepares a TASU shared interface for pure text alignment for speech understanding in the multilingual speech recognition model. In the TASU shared interface, the posterior provider, CTC output space, large language model (LLM), target languages, datasets, and projectors are shared. This interface eliminates language-specific alignment heads and unifies the training and inference input interfaces.
[0022] For the target languages and datasets in this method, the target languages are the eight languages of the multilingual LibriSpeech dataset: Dutch, English, French, German, Italian, Polish, Portuguese, and Spanish (Nl / En / Fr / De / It / Pl / Pt / Es). However, the original MLS training set data distribution is extremely imbalanced: the number of speech samples for Dutch, English, French, German, and Spanish is far greater than that for Italian, Polish, and Portuguese. To obtain a more balanced training set, the upper limit for samples in each high-resource language is set to 196,000, while retaining almost all available data for the three low-resource languages. This sampling strategy ultimately yields approximately 1.1 million speech samples, accounting for about one-third of the complete MLS training set. This proportion comes from balanced sampling processing for language balancing, not uniform downsampling. Figure 3 The processed data distribution is presented. All training experiments were constructed based on this balanced dataset. In specific implementations, the real posterior experimental group used the OWSM-CTC posterior obtained from the corresponding speech extraction; the heuristic CPS (CTC Posterior Simulation) experimental group used only the transcribed text. For the learning simulator experimental group, a lightweight quality filter was used to remove a small number of obviously anomalous posterior sequences, including extremely long samples and outputs with excessively high blank ratios. In the few-pair speech experiments, the complete balanced transcribed text set was used during the text initialization stage, followed by a round of real posterior adaptation on 1 / 3 of the corresponding speech subset, which is approximately 1 / 9 of the MLS complete training set. Validation and testing for all 8 languages used the official MLS partitioned dataset.
[0023] For large language models (LLMs), Qwen, Kimi, etc., can be selected. In the subsequent steps of this implementation, the Qwen2.5-1.5B-Instruct large language model is selected. Furthermore, a single speech placeholder can be set in the prompt template of the large language model. <speech>The projected variable-length posterior embedding sequence The speech placeholders are replaced to form the language model input sequence, so as to share the cross-language alignment capability of the interface and avoid the parameters from increasing with the number of languages.
[0024] The posterior provider in this method can be OWSM-CTC (Open Whisper-style Speech Model with CTC) v4 1B. The posterior provider uses the same V-class output space for all target languages, with V classes including a blank class. The output space contains 50,002 classes, with the blank class index at 0. Alternatively, the posterior provider can employ other CTC front-ends trained to provide a shared CTC output space for multiple target languages.
[0025] For posterior projectors whose shared output posterior distributions lie in the same connected temporal classification vocabulary space, this method establishes a cross-linguistic shared posterior projector. Each posterior frame is mapped to a large language model hidden dimension H. This embodiment uses a single-layer bottleneck-free affine mapping, which can map a 50,002-dimensional posterior to a 1,536-dimensional language model input embedding.
[0026] As one implementation, the multilingual speech recognition model also includes shared low-rank adaptation parameters, meaning it also benefits from the sharing among TASU shared interfaces.
[0027] Specifically, such as Figure 4 A general model configuration is summarized. A frozen OWSM-CTC v4 1B model is used as the multilingual acoustic posterior generator. Its CTC output has 50,002 classes, with index 0 representing the blank class. Each speech sample input is accompanied by the corresponding OWSM language identifier and ASR task symbol. Subsequently, label-synchronized posterior compression is performed, with a blank threshold set to 0.9. Consecutive frames with the same non-blank label obtained from argmax are averaged. OWSM-CTC remains frozen throughout TASU training, thus pre-calculating the posterior for the audio in one go, avoiding repeated runs of the acoustic front-end during training iterations. The probability distribution of the CTC posterior is typically highly concentrated; therefore, only the 10 most probable values and their corresponding token indices are retained for each compressed frame. The projector directly operates based on this sparse representation, significantly reducing data loading overhead and computational cost associated with the 50,002-dimensional input. The retained values are the original OWSM probabilities, without re-normalization; probabilities outside the top-10 are directly set to zero. This offline sparse representation is used only for training; during the testing phase, the OWSM-CTC posterior is calculated directly from the input speech in real time.
[0028] The above implementation methods demonstrate the unique architecture of the multilingual speech recognition model in this method, which includes a posterior simulator for predicting connection temporal classification, a shared frozen posterior provider for multilingual connection temporal classification, a projector for cross-language connection temporal classification sharing, a large language model with a shared interface, and a multilingual training set obtained by balanced sampling of different target languages to train the multilingual speech recognition model.
[0029] For step S12, the first stage involves training the shared projector. The intention behind this training is to utilize massive amounts of readily available plain text data to complete the cross-modal alignment initialization of the shared projector and LoRA without using real audio-text pairing samples. This teaches the shared projector to map the CTC-formatted posterior to an embedding that the large model can understand, and allows LoRA to learn the mapping relationship between posterior features and the large model's text semantic space in different languages. This ensures that the simulated posterior generated from the text, after passing through the projector and LoRA, is equivalent to the text embedding of the original text. In other words, the first stage of this method contains no audio whatsoever; it addresses the modality switching problem.
[0030] As one implementation, the posterior simulator is used to predict connection time series classification, including: a heuristic connection time series classification posterior simulator or a multilingual learning simulator, which establishes an initial alignment from the posterior to the LLM by using a large amount of text.
[0031] In this embodiment, a multilingual simulated posterior can be generated using CPS or a frozen simulator. Optionally, during training, cross-language shared LoRA parameters are added to the attention projection layer and feedforward projection layer of the large language model, and trained together with the posterior projector. In this embodiment, the LoRA rank is 64, the scaling parameter is 16, and the dropout is 0.05.
[0032] As one implementation, when the posterior simulator is a heuristic connectionist temporal classification posterior simulator, the step of generating a multilingual simulated posterior through the posterior simulator includes: The text is transcribed and encoded into a connection time-series classification category sequence; For each category sequence, construct a smooth posterior and sample confidence probabilities; Delete the removable category posterior frames and insert additional frames according to the stated confidence probability to obtain a heuristic multilingual simulated posterior.
[0033] In this embodiment, the multilingual text transcription is encoded into a CTC category sequence y=(y1,…,yU). Then, a smoothed one-hot posterior is constructed for each category, and a confidence level α is sampled for the entire sequence. Where δ(yi) is the one-hot vector of category yi, 1 / V is a uniform distribution, and α is sampled from the uniform distribution in the interval [λlow, λhigh].
[0034] With probability Independently delete removable category posterior frames, then sort them according to probability. Insert additional frames, which are copies of the posterior of adjacent classes or blank one-hot posteriors. Specifically, this method can sample the entire sequence α∈[0.8,1.0] and set... = =0.05; the same simulation parameters are used for each target language. Through the above method, a heuristic multilingual simulation posterior is obtained.
[0035] As one implementation, when the posterior simulator is a multilingual learning simulator, pre-training of the posterior simulator is performed before the first training phase, including: Generating true teacher posteriors using a posterior provider to produce paired text-speech samples within multilingual training sets; The predicted posterior of the text is determined using the multilingual learning simulator. The multilingual learning simulator is trained using only the cross-entropy loss between the real teacher posterior and the predicted posterior, and then frozen after training is completed.
[0036] Heuristic simulation of the posterior may fail in multilingual native spaces. Therefore, to address this issue, this method pre-trains a simulator for multilingual text-to-posterior transformation. Conceptually, because low-resource languages cannot construct data by WER bucketing, the method does not use TASU2's binning and data augmentation, but instead generates data based on single conditions and performs quality filtering on ill-conditioned outputs.
[0037] In this embodiment, a frozen multilingual CTC posterior provider is used to generate a true teacher posterior P=(p1,…,pT) for the multilingual paired speech, and the corresponding text category sequence of the paired samples is used as the simulator input. The simulator predicts a variable-length posterior sequence based on the text. Target posterior completion to training length Valid frames are identified using a mask `mt`, and padded positions are not included in the loss calculation. The simulator is trained using posterior-level masked cross-entropy. in, For simulator loss function, For simulator network parameters, To calculate the expected value of the training samples, This is a frame-level mask; valid frames are set to 1, and padded frames are set to 0. Padded frames are not included in the loss calculation. For the teacher's true posterior, the probability of the v-th class in the t-th frame. The predicted output posterior for the simulator is the predicted probability of the v-th class in the t-th frame.
[0038] This loss enables the simulator to learn the duration, label confusion, blank distribution, and contextual dependencies in the real posterior. This embodiment employs an autoregressive Transformer with a model dimension of 512, containing 8 attention heads, 6 encoder layers, 6 decoder layers, and a 2048-dimensional feedforward layer, generating the posterior with a temperature of 0.5. After the posterior simulator training is complete, it is frozen, and a simulated posterior is generated offline from multilingual text for downstream TASU alignment training. No waveform is input during this downstream alignment stage; the simulator's pre-training uses the real teacher posterior extracted from paired speech.
[0039] Regarding the training method, text transcription y is used as output supervision, and TASU optimizes by minimizing the output under teacher-mandated conditions: Given an input posterior sequence P and a transcribed text The projector maps each posterior frame to the Qwen embedding space and outputs... The sequence replacement instruction prompt obtained after projection <speech>Placeholder. In teacher-mandated training mode, where 'q' represents a text instruction. This is the end-of-sequence token. The prompt word and its corresponding position in the posterior are not included in the loss calculation. All eight languages share the same set of posterior projectors, Qwen backbone network, and LoRA parameters. No language-specific projectors or output heads are introduced. In this way, by performing modality switching optimization training on the projectors through the above multilingual simulated posterior, a shared cross-language aligned projector is obtained.
[0040] This calculation is equivalent to: scattering and filling the retained probability values into a 50002-dimensional zero vector, and then performing a dense affine layer operation. The projected sequence replaces a single word in the Qwen prompt. <speech>Placeholder.
[0041] For step S13, in the second training phase, the posterior provider is used to determine the true posterior of another text-speech pairing sample within the multilingual training set. The purpose of the second phase is to correct the distribution offset between the simulated posterior and the true acoustic posterior introduced in the first phase. The first phase learns an ideal virtual posterior distribution; this phase uses a small amount of real audio to generate the true posterior, pulling the parameters towards the distribution of real acoustic data, solving the distribution mismatch problem. Through true posterior processing, the sequence is shortened, and blanks and redundant labels are reduced. Lightweight fine-tuning calibration is performed based on the cross-language alignment already learned in the first phase, without needing to randomly initialize training from scratch. Using only a small amount of paired speech, a model adapted to real audio input can be obtained, alleviating the problem of scarce multilingual paired speech resources. After training is complete, a final usable model is obtained, which can be used for inference: receiving real audio for multilingual speech recognition. In summary, existing technologies use a training method of initial training in the first phase and fine-tuning in the second phase, but the conventional pre-training and fine-tuning phases of existing technologies use data of the same modality and only address different domains or tasks. The first stage of this method contains no audio whatsoever; the input is a simulated posterior generated from text. The second stage switches to a real audio posterior. This method addresses modality switching and distribution mismatch issues in both stages, not domain adaptation.
[0042] The second stage loads the projector and LoRA weights from the text initialization checkpoint, and continues calibration using a small amount of paired speech data generated by the frozen CTC posterior provider. In this embodiment, CPS initialization is performed using the complete balanced text set, and then a true posterior adaptation round is performed using approximately one-ninth of the data from the complete MLS paired speech training set.
[0043] Specifically, after the second phase of training begins, the text initialization checkpoints trained in the first phase are loaded first. The projector weights and LoRA adapter weights stored in these checkpoints are read and loaded, and the core parameters of the large language model are kept frozen throughout the process without any updates. In this phase, a small number of multilingual audio-text pairing samples are used for training. In practice, the speech data of the samples and the corresponding language identifiers are directly input into the frozen multilingual CTC posterior provider, and the model infers and generates a real teacher posterior. Next, this set of real teacher posteriors is fed into the projector with initialized weights and converted into a frame-level speech feature sequence. Then, the system's preset fixed task prompt template is retrieved to generate the corresponding original embedding sequence. The speech placeholder positions in the sequence are located, and the converted speech feature sequence is used to replace the placeholder embeddings. The resulting sequence is then concatenated to form a mixed input embedding sequence, which is then input into the large language model loaded with the LoRA adapter. The training supervision is based on the manually transcribed text that comes with the pairing samples, which is used to calculate the model's autoregressive loss. When updating parameters during backpropagation, only the parameters of the projector and LoRA adapter are fine-tuned, while the core parameters of the large language model remain frozen. The entire calibration process is completed by iteratively training all a small number of paired samples. After training, the calibrated and optimized projector and LoRA weights are saved to obtain the final adapter weights that can adapt to real audio input. Real audio is then received to complete the multilingual speech recognition task. Simply put, in the two-stage training, only the projector can be trained, or both the projector and LoRA can be trained simultaneously. Training only the projector may result in uneven improvement for some languages. Training both the projector and LoRA together ensures stable and uniform improvement for all languages. In practice, the choice can be made based on factors such as the number of languages.
[0044] As one implementation, determining the true posterior of another text-speech pairing sample within the multilingual training set using the posterior provider includes: The paired samples are input into the posterior provider to obtain the true posterior sequence at the frame level; The blank frames of the true posterior are filtered out and compressed so that the structure of the compressed true posterior sequence approximates the category sequence of the text.
[0045] In this embodiment, the speech, corresponding language symbols, and automatic speech recognition task symbols in the paired samples are input into the frozen multilingual CTC posterior provider to obtain... Frame-level true posterior.
[0046] Then delete frames with a blank probability higher than the threshold τ: Group consecutive frames with the same non-blank maximum probability label from the retained sequence into a set Sj, and average the within-group posterior: This embodiment uses τ=0.9. Posterior compression is performed by first deleting the dominant blank frame, then merging consecutive frames with the same label in the retained sequence. The compressed posterior sequence is shorter and its structure is closer to the text category sequence.
[0047] The trainable projector directly maps the compressed posterior to the Qwen2.5-1.5B-Instruct model input space. For models retaining indices... The corresponding probability is For sparse frames, the model calculates: in, For the weight matrix of the shared posterior projector, Indicates the first The weight row vectors corresponding to each category, with b as the bias. This result is the same as the result of first scattering the Top-K probabilities to a V-dimensional zero vector and then performing a complete affine mapping, which is equivalent to dense affine projection and reduces the burden of training data loading and computation.
[0048] Compression using the above method yields a category sequence that approximates text, with the specific structure as follows: Figure 5 As shown.
[0049] As one implementation method, after obtaining the trained multilingual speech recognition model, Using the aforementioned multilingual speech recognition model to perform multilingual real speech recognition inference includes: The multilingual speech recognition model receives the real speech to be recognized and the target language identifier; use The posterior distribution of the output is in the same connected temporal classification vocabulary space. The posterior provider generates the true posterior online. The blank frames after removing the true posterior are compressed. The processed true posterior is input into a shared cross-language aligned projector that optimizes modality switching and distribution mismatch, resulting in an embedding sequence with the same hidden dimension as the large language model. The embedded sequence is converted into speech recognition results in the target language using the large language model.
[0050] In this embodiment, the system receives the speech to be recognized, the target language identifier, and the automatic speech recognition task identifier. A true CTC posterior is generated online using a frozen multilingual CTC posterior provider. The posterior generated by CPS or a learning simulator is not used during the inference phase. Blank frame filtering, merging of consecutive frames with the same non-blank label, and optional Top-K sparsification are performed on the true posterior. The processed true posterior is then input into the same trained projector, and the resulting embedding sequence replaces the prompts in the large language model. <speech>Placeholder. The target language transcription result is generated by autoregression from a large language model. Under this process, the following is achieved: Figure 6 The overall structure of the above steps in this method is shown.
[0051] In summary, this method includes a text-driven training path, a true posterior calibration path, and a speech inference path. For example... Figure 7 As shown, all three paths operate within the same multilingual CTC posterior space and reuse the same posterior projector and large language model. The simulated posterior generated by the text-driven path is used for training or initialization, while the speech inference path always uses the true posterior generated from the input speech. Specifically, the steps include: selecting a posterior provider capable of providing a shared native CTC vocabulary for multiple target languages and freezing this posterior provider during alignment training; segmenting the text of each language according to the CTC vocabulary and generating simulated posteriors in the same V-dimensional space using a heuristic CPS or a trained and frozen multilingual posterior simulator; inputting the simulated posterior into the cross-language shared posterior projector to obtain an embedding sequence with the same hidden dimension as the large language model, and replacing the speech placeholders in the prompt words; training the projector using the corresponding transcribed teacher-forced loss and adding cross-language shared low-rank adaptation parameters to the attention and feedforward layers of the same large language model; after text initialization, using the true posterior generated from a small number of paired speech samples to continue calibrating the projector and low-rank adaptation parameters; this embodiment performs one round of true posterior adaptation. During inference, the input speech is converted into a real CTC posterior online. After blank frame filtering, merging of consecutive same-label frames, and optional Top-K sparsification, the same projector and large language model used in the training phase are used to generate the transcription.
[0052] As can be seen from this implementation, the method establishes a unified cross-language interface, allowing multiple target languages to project within the same native CTC posterior space, using a large language model and LoRA parameters. This eliminates the need for separate alignment modules for each language, reducing reliance on paired speech and improving multilingual recognition performance. Furthermore, it enhances the quality of the simulated posterior, reduces the burden of posterior access and projection, and improves training reusability, making data balancing and language coverage executable data preparation steps.
[0053] Further details regarding the training process are provided. The encoder generalization experiment used Whisper-Large v3 with a randomly initialized CTC head, and the output classes were aligned to the Qwen2.5 1.5B Instruct vocabulary (151,667 classes). Due to limitations in the training data size, this large vocabulary was difficult to converge sufficiently; therefore, a reduced output head with 38,243 classes was additionally tested. The training dataset for the CTC head was identical to the balanced MLS subset used in the OWSM-CTC experiments. The CTC head employed the AdamW optimizer with a peak learning rate of [missing information]. The training process consisted of 1000 steps, followed by cosine decay. Training precision was bfloat16, based on eight Ascend NPUs using DeepSpeed ZeRO 2; dynamic batch processing was enabled, with a frame rate cap of 3200, and no gradient accumulation was performed. The training consisted of three epochs with a random seed of 42. All experimental systems shared the same OWSM CTC (Open Whisper-style Speech Model with CTC) posterior space, posterior projector, and language model; the differences lay in the source of posterior supervision used during training and whether the language model employed LoRA (Low-Rank Adaptation). During inference, all systems used the true OWSM CTC posterior obtained from the input speech.
[0054] Regarding experimental configuration, this method divides experimental groups according to the posterior source used in the TASU adaptation phase. Real: Posterior extracted offline from MLS paired audio using OWSM-CTC. CPS: Heuristic pseudo-posterior constructed directly from transcribed text. Simulator: Loaded with a frozen multilingual TASU2 simulator to predict the posterior for OWSM-segmented text. The simulator is trained solely using the real teacher posterior; therefore, "text only" in this group refers to the downstream TASU adaptation phase, not the complete simulator training chain. During the testing phase, all TASU systems receive the same real posterior generated in real-time by the same OWSM-CTC frontend. Projector-only experimental group: The OWSM frontend and Qwen backbone network are completely frozen; only the posterior projector is optimized. LoRA experimental group: The projector remains trainable, while LoRA parameters are introduced into the Qwen attention projection layers (q, k, v, o) and feedforward network projection layers (gate, up, down). Two-stage Experimental group: Initialize with checkpoints obtained from text training, then train for an additional epoch using a small subset of real posteriors. Simulator coverage ablation experiment: Only remove Polish teacher posterior pairs from the simulator training set; the resulting 7-language simulator still generates posteriors for all 8 languages, and the downstream TASU model is still trained on all 8 languages. In addition to the main experiment, replace with the Whisper-CTC frontend to reproduce 3 core training configurations (real posterior, audio only, ...). (Through course learning) this study to verify that the performance improvement brought by TASU is not exclusive to the OWSM-CTC model.
[0055] This method uses a multilingual speech recognition model based on posterior and text simulation, trained with the AdamW optimizer, and a peak learning rate of [value missing]. A 700-step learning rate warm-up was performed, followed by cosine decay. Training precision was bfloat16, based on 16 Ascend NPUs, with DeepSpeed ZeRO 2 enabled. In each optimizer step, each device processed one dynamically assembled sample packet; constrained by the configuration length limit, each sample packet contained a maximum of 4 speech samples. Gradient accumulation was not used. The model was trained for a maximum of 3 epochs, with a maximum of 55,000 optimizer steps. All experiments used a fixed random seed of 42. The offline real posterior and the simulator-generated posterior employed the exact same sparse representation and data loading process, ensuring complete consistency in the downstream projector's computation process.
[0056] To supplement the encoder generalization experiment, another front-end was built: the acoustic encoder weights were initialized from Whisper-Large-v3, and a randomly initialized CTC head was trained on the same balanced MLS subset. The CTC head vocabulary was aligned with Qwen2.5-1.5B-Instruct, and the output layer dimension was 150,000. Due to the limitation of the training data size, the parameters of this large vocabulary were difficult to estimate reliably; therefore, the additional test vocabulary was reduced to a 38k CTC head. All downstream TASU training and evaluation configurations (optimizer, learning rate, LoRA hyperparameters, decoding strategy) were completely consistent with the OWSM CTC experimental group, achieving a controllable comparison.
[0057] When decoding the official MLS test sets, the corresponding OWSM language symbols are input. The reference text and model output text are Unicode normalized and lowercaseed, ASR punctuation is removed, and apostrophes within words are retained. WER (Word Error Rate) is implemented using the WeNet evaluation script. The overall metric is obtained by summing all reference words from the eight test sets, rather than performing an unweighted average of the WERs for each of the eight languages.
[0058] Based on the above experimental setup, the following experimental results were obtained: like Figure 8 As shown, the experimental results of different posterior sources are compared when only the posterior projector is trained. The WER obtained by training with the real OWSM posterior is almost on par with the original OWSM CTC; the performance improvement is not uniform: French and Portuguese show significant improvement, but the Polish WER deteriorates from 15.09 to 23.55. Based on the learned simulator posterior training, the overall WER is 11.50, which is far better than the heuristic CPS scheme (20.55); however, the results of both simulator posterior experimental groups are not as good as those trained with the real posterior. These results indicate that the multilingual shared posterior interface scheme is feasible; however, the distribution mismatch between the text-generated posterior and the real acoustic posterior remains the main bottleneck; training only the projector between the CTC posterior and the large model is insufficient to fully utilize the model's performance.
[0059] like Figure 9 As shown, the experimental results for each group of experiments using LoRA to fine-tune the Qwen backbone network in addition to training the posterior projector are presented. When using the real posterior, the overall WER of TASU Real LoRA reaches 8.21, a relative decrease of 18.6% compared to OWSM CTC. Unlike the real posterior approach that only trains the projector, this approach achieves performance improvements for all eight languages, with particularly significant absolute gains for French, Italian, and Portuguese. For TASU trained based on the simulator posterior, LoRA also brings performance improvements, reducing the overall WER from 11.50 to 10.51, but its performance is still weaker than OWSM CTC and the real posterior adaptation approach. In summary, fine-tuning large models can better utilize both types of posterior sources, but it cannot eliminate the distribution mismatch between the simulator-generated posterior and the real acoustic posterior.
[0060] In two phases The system, using only 1 / 9 of the complete MLS paired speech dataset, achieved an overall WER of 9.19 after one round of real posterior training; this represents an 8.8% decrease compared to the original OWSM CTC, with WER reductions observed in 7 out of 8 languages, except for Polish. Under these deliberately limited adaptation data conditions, the text-initialized model achieved competitive results after only one round of real posterior training. This result demonstrates the practical advantages of the text-first TASU approach: it can first establish multilingual alignment relationships using a large number of text transcripts, and then utilize a small number of paired speech samples for supervision, resulting in a high-performing multilingual speech recognition model.
[0061] like Figure 10 As shown, this study explores whether the trained posterior simulator can generalize to languages not present in the training set. In this ablation experiment, Polish teacher posterior pairs were removed from the simulator training data, and the simulator was trained only on seven languages. These seven-language simulators were then used to generate CTC posteriors for transcribed texts in all eight languages (including Polish). The downstream TASU model was still trained using simulator posteriors for all eight languages; Polish was excluded from simulator training but still participated in TASU training. After removing Polish from simulator training, the other seven languages seen by the simulator were almost unaffected, with the macro-average WER decreasing from 10.79 to 10.61. However, the Polish WER surged from 16.38 to 80.47. This performance degradation was not due to the lack of Polish text in TASU training; rather, the simulator, having not seen Polish teacher posteriors during training, could not output a valid Polish CTC posterior, and the downstream projector and LoRA parameters could not compensate for this distribution mismatch. This ablation experiment demonstrates that even if TASU can be trained using texts from all target languages, the simulator training phase still needs to cover the target languages.
[0062] Based on the above experiments, extending the posterior-based alignment scheme beyond English requires meeting two requirements. First, a single shared TASU model must be able to stably optimize the posterior of real multilingual CTC models, as evidenced by the performance gains of TASU Real LoRA across languages. Second, the training method based on text generation posterior is constrained by two factors: there is a distributional mismatch between the simulator training posterior and the real posterior during inference, and the language coverage during simulator training also limits performance. The experiment that only removed Polish cannot directly prove that all language pairs are completely independent; however, the result shows that even if the downstream TASU is trained using transcribed text in a certain language, the simulator may not be able to generalize to languages not seen in the simulator's training set. To verify whether the above conclusions apply only to the OWSM CTC encoder, the following section replaces the TASU model with the Whisper CTC frontend and reproduces the core training configuration. The previous experiments used OWSM-CTC as the posterior output module to build the TASU model. To demonstrate that the performance improvement is not limited to this encoder, this paper uses the Whisper-CTC frontend to reproduce the core training configuration. This encoder is built on Whisper-Large-v3, with CTC header parameters randomly initialized, and is trained on a small multilingual subset. Its output posterior distribution differs from the former, and its baseline word error rate (WER) is significantly higher.
[0063] like Figure 11 This indicates that the two-stage Phase1→Phase2 learning strategy can be effectively transferred to the Whisper-CTC frontend. Compared to the native Whisper-CTC, the WER of this approach decreased for all eight languages; it outperformed audio-only TASU (except for Dutch) for seven languages, with the aggregate WER decreasing from 17.62 to 14.67, a relative reduction of 16.7%. The Phase1 stage alone performed poorly: the model used text-generated posteriors for training, but input the true Whisper-CTC posterior during inference. After one round of adaptation to the true posterior, the aggregate WER decreased from 35.54 to 14.67. Combined with the experimental results on OWSM-CTC, it can be concluded that the text-first initialization strategy can effectively adapt to the true posterior input from different CTC frontends; however, the superiority of "audio-only training" versus "two-stage training" depends on the posterior output module and adaptation conditions. Reducing the vocabulary of the Whisper-CTC head can optimize the aggregate WER of the native model from 25.91 to 24.42. However, all subsequent TASU experiments used the complete vocabulary header. This comparison is only used for front-end feature analysis and cannot directly prove that vocabulary size affects TASU itself. This method also provides a 30k-sized CTC header as a reference. Reducing the vocabulary from 150k to 30k resulted in a decrease in the native Whisper-CTC baseline WER from 25.91 to 24.42. Subsequent calculations of TASU performance gains consistently used the 150k large vocabulary baseline as the benchmark, thus separating the benefits from alignment quality from those from vocabulary design. In summary, repeated experiments on Whisper-CTC demonstrate that multilingual TASU can generalize to different CTC encoders. Course learning brings stable relative improvements, and the performance ranking among different training configurations remains unchanged, proving that TASU is a general multilingual alignment framework, rather than a customized method only adaptable to specific acoustic posterior output modules.
[0064] In summary, this method leverages the native posterior space of OWSM CTC, allowing TASU to be extended from English into a single-shared-weights multilingual speech model. Adapting the projector to the large language model using the real posterior space results in stable improvements in recognition performance for all eight languages. The posterior space generated from text provides a practical and effective initialization for subsequent speech adaptation. Specifically, the trained multilingual simulator significantly outperforms the heuristic CPS method; the text-initialized TASU model requires only a small amount of real posterior supervision for efficient calibration. These results validate a practical multilingual training paradigm: first, establish the alignment between the posterior space and the large model using a massive amount of text, then adapt the model to the real acoustic posterior space using a small amount of paired speech.
[0065] like Figure 12 The diagram shows a schematic of a multilingual speech recognition model training agent system based on posterior and text simulation provided in an embodiment of the present invention. The system can execute the multilingual speech recognition model training method based on posterior and text simulation described in any of the above embodiments and is configured in a terminal.
[0066] This embodiment provides a multilingual speech recognition model training agent system 10 based on posterior and text simulation, which includes a speech recognition module 11 and a training module 12.
[0067] The speech recognition module 11 is used to input the multilingual training set into the multilingual speech recognition model. The multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters across multiple languages, and a large language model. The training module 12 is used in the first training stage to generate a multilingual simulated posterior using only the text in the multilingual training set through the posterior simulator, and to perform modality switching optimization training on the projector through the multilingual simulated posterior to obtain a projector with shared cross-language alignment. In the second training stage, the true posterior is determined by using a text-speech pairing sample from another text in the multilingual training set through the posterior provider, and the projector distribution mismatch optimization training is continued through the true posterior to obtain the trained multilingual speech recognition model.
[0068] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the multilingual speech recognition model training method based on posterior and text simulation in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: The multilingual training set is input into the multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters across multiple languages, and a large language model; In the first training phase, multilingual simulated posteriors are generated using only the text in the multilingual training set through the posterior simulator. The projector is then trained using modality switching optimization through the multilingual simulated posteriors to obtain a projector with shared cross-language alignment. In the second training phase, the true posterior is determined by the posterior provider using a paired text-speech sample from another text-speech sample in the multilingual training set. The projector distribution mismatch is then optimized and trained using the true posterior to obtain the trained multilingual speech recognition model.
[0069] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the multilingual speech recognition model training method based on posterior and text simulation in any of the above method embodiments.
[0070] Figure 13 This is a schematic diagram of the hardware structure of an electronic device based on a posterior and text simulation multilingual speech recognition model training method provided in another embodiment of this application, as shown below. Figure 13 As shown, the device includes: One or more processors 1310 and memory 1320, Figure 13 Taking a processor 1310 as an example, the device for training a multilingual speech recognition model based on posterior and text simulation may further include an input device 1330 and an output device 1340.
[0071] The processor 1310, memory 1320, input device 1330, and output device 1340 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.
[0072] The memory 1320, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multilingual speech recognition model training method based on posterior and text simulation in the embodiments of this application. The processor 1310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1320, thereby implementing the multilingual speech recognition model training method based on posterior and text simulation in the above-described method embodiments.
[0073] The memory 1320 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1320 may optionally include memory remotely located relative to the processor 1310, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0074] Input device 1330 can receive input numerical or character information. Output device 1340 may include display devices such as a display screen.
[0075] The one or more modules are stored in the memory 1320, and when executed by the one or more processors 1310, they execute the multilingual speech recognition model training method based on posterior and text simulation in any of the above method embodiments.
[0076] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.
[0077] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0078] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multilingual speech recognition model training method based on posterior and text simulation according to any embodiment of this invention.
[0079] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.
[0080] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.
[0081] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0082] (4) Other electronic devices with data processing functions.
[0083] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / speech> < / speech> < / speech> < / speech>
Claims
1. A method for training a multilingual speech recognition model based on posterior and text simulation, comprising: The multilingual training set is input into the multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters across multiple languages, and a large language model; In the first training phase, multilingual simulated posteriors are generated using only the text in the multilingual training set through the posterior simulator. The projector is then trained using modality switching optimization through the multilingual simulated posteriors to obtain a projector with shared cross-language alignment. In the second training phase, the true posterior is determined by the posterior provider using a paired text-speech sample from another text-speech sample in the multilingual training set. The projector distribution mismatch is then optimized and trained using the true posterior to obtain the trained multilingual speech recognition model.
2. The method according to claim 1, characterized in that, The multilingual speech recognition model also includes: shared low-rank adaptation parameters; In the first training phase, multilingual simulated posteriors are generated using only the text in the multilingual training set through the posterior simulator. The projector is then trained for modality switching optimization using the multilingual simulated posteriors. During training, the shared low-rank adaptation parameters are added to the attention projection layer and feedforward projection layer in the large language model to obtain a jointly trained cross-lingual aligned projector and low-rank adaptation parameters. In the second training phase, the true posterior is determined by the posterior provider using another text-speech pairing sample in the multilingual training set. The projector and the low-rank fitting parameter distribution mismatch optimization training are then continued using the true posterior to obtain the trained multilingual speech recognition model.
3. The method according to claim 1, characterized in that, The posterior simulator is used to predict connection time series classification, including: a heuristic connection time series classification posterior simulator or a multilingual learning simulator.
4. The method according to claim 3, characterized in that, When the posterior simulator is a heuristic connectionist temporal classification posterior simulator, the step of generating a multilingual simulated posterior through the posterior simulator includes: The text is transcribed and encoded into a connection time-series classification category sequence; For each category sequence, construct a smooth posterior and sample confidence probabilities; The removable category posterior frames are deleted, and additional frames are inserted according to the stated confidence probability to obtain a heuristic multilingual simulated posterior.
5. The method according to claim 3, characterized in that, When the posterior simulator is a multilingual learning simulator, pre-training of the posterior simulator is performed before the first training phase, including: Generating true teacher posteriors using a posterior provider to produce paired text-speech samples within multilingual training sets; The predicted posterior of the text is determined using the multilingual learning simulator. The multilingual learning simulator is trained using only the cross-entropy loss between the real teacher posterior and the predicted posterior, and then frozen after training is completed.
6. The method according to claim 1, characterized in that, Determining the true posterior through the posterior provider includes: The paired samples are input into the posterior provider to obtain the true posterior sequence at the frame level; The blank frames of the true posterior sequence are filtered out and compressed so that the structure of the compressed true posterior sequence approximates the category sequence of the text.
7. The method according to claim 1 or 2, characterized in that, After obtaining the trained multilingual speech recognition model, multilingual real speech recognition inference is performed using the multilingual speech recognition model, including: The multilingual speech recognition model receives the real speech to be recognized and the target language identifier; The true posterior is generated online using a posterior provider whose output posterior distribution is in the same connected temporal classification vocabulary space; The blank frames after removing the true posterior are compressed. The processed true posterior is input into a shared cross-language aligned projector that optimizes modality switching and distribution mismatch, resulting in an embedding sequence with the same hidden dimension as the large language model. The embedded sequence is converted into speech recognition results in the target language using the large language model.
8. A multilingual speech recognition model training system based on posterior and text simulation, comprising: A speech recognition module is used to input multilingual training sets into a multilingual speech recognition model, wherein the multilingual speech recognition model includes: a posterior simulator and a posterior provider whose output posterior distributions are in the same connection-temporal classification vocabulary space, a projector that shares the same parameters for multiple languages, and a large language model; The training module is used in the first training phase to generate a multilingual simulated posterior using only text from the multilingual training set through the posterior simulator, and to perform modality switching optimization training on the projector using the multilingual simulated posterior to obtain a projector with shared cross-language alignment. In the second training phase, the module uses a text-speech pairing sample from another text in the multilingual training set to determine the true posterior using the posterior provider, and continues to optimize the projector distribution mismatch training using the true posterior to obtain the trained multilingual speech recognition model.
9. The system according to claim 8, characterized in that, The multilingual speech recognition model training system also includes an inference module, used for: The multilingual speech recognition model receives the real speech to be recognized and the target language identifier; The true posterior is generated online using a posterior provider whose output posterior distribution is in the same connected temporal classification vocabulary space; The blank frames after removing the true posterior are compressed. The processed true posterior is input into a shared cross-language aligned projector that optimizes modality switching and distribution mismatch, resulting in an embedding sequence with the same hidden dimension as the large language model. The embedded sequence is converted into speech recognition results in the target language using the large language model.
10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.
11. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.
12. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.
