Speech recognition large model training method and device, storage medium and equipment

Abstract

The invention relates to the technical field of artificial intelligence, and provides a training method and device for a voice recognition large model, a medium and equipment, and the method comprises the steps: obtaining a training data set; inputting the training data set into the initial large model, and performing identification processing on the audio sample through a streaming identification branch in the initial large model to obtain a first candidate text set; according to the text consistency between the target candidate text in the first candidate text set and the real labeled text, determining whether to activate a non-streaming recognition branch to generate a second candidate text set; and performing iterative training on the initial large model according to the semantic difference between each second candidate text and the real labeled text and the text length difference between each second candidate text and the target candidate text to obtain a trained united streaming and non-streaming speech recognition large model. According to the method and the device, the problem of subtitle jumping caused by secondary identification length change can be relieved while the identification accuracy is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a training method for a large speech recognition model, a training device for a large speech recognition model, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Automatic Speech Recognition (ASR) training, a core component for human-computer voice interaction, is widely used in scenarios such as online conferencing, real-time caption generation, in-vehicle navigation, and intelligent assistants. In many applications, the system needs to convert speech streams into text in real time with low latency, which involves training a streaming speech recognition model. Because streaming recognition processes signals segment by segment during speech input, lacking complete contextual information, its recognition accuracy is typically lower than that of non-streaming (offline) recognition, which can utilize global context.

[0003] To improve the final accuracy of streaming recognition, existing technologies often employ a two-pass recognition architecture: the first pass uses a streaming model to achieve low-latency output, meeting real-time requirements; the second pass, after sentence end or silence detection (such as Voice Activity Detection, VAD), utilizes the complete speech segment for refined non-streaming re-recognition to correct the initial result. However, while this post-processing correction mechanism improves accuracy, it introduces a significant user experience problem—subtitle skipping.

[0004] Therefore, how to effectively alleviate the problem of subtitle jumping caused by the inconsistency in the length of the two recognition results while ensuring the improvement of recognition accuracy has become a key challenge that current streaming speech recognition technology urgently needs to solve.

[0005] Therefore, there is an urgent need in this field to develop a new training method and apparatus for large-scale speech recognition models.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0007] The purpose of this disclosure is to provide a training method for a large speech recognition model, a training device for a large speech recognition model, a computer-readable storage medium, and an electronic device, thereby overcoming, to at least a certain extent, the technical problem of being unable to improve recognition accuracy while alleviating subtitle jumping due to limitations in related technologies.

[0008] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0009] According to a first aspect of this disclosure, a method for training a large speech recognition model is provided, comprising: Obtain a training dataset consisting of audio samples and their corresponding real-text annotations; The training dataset is input into the initial large model, and the audio samples are processed through the streaming recognition branch to obtain the first candidate text set; Based on the text consistency between the target candidate text in the first candidate text set and the real labeled text, determine whether to activate the non-streaming recognition branch to generate the second candidate text set. Based on the semantic difference between each second candidate text and the real labeled text, and the text length difference between each second candidate text and the target candidate text, the initial large model is iteratively trained to obtain a trained joint streaming and non-streaming speech recognition large model.

[0010] In an exemplary embodiment of this disclosure, the streaming identification branch includes a streaming encoder and a connected temporal classification decoder; The step of obtaining a first candidate text set by recognizing the audio samples through the streaming recognition branch includes: The audio samples are encoded using the streaming encoder to generate streaming encoded features; The streaming encoding features are input into the connection-time classification decoder to decode and obtain a first candidate text set containing multiple first candidate texts and their corresponding confidence scores.

[0011] In an exemplary embodiment of this disclosure, the initial large model further includes a decision branch; the decision branch includes a projector and a pre-trained large language model; The step of determining whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text includes: Multiple first candidate texts and their confidence scores are concatenated according to preset rules to form a streaming recognition sequence; The streaming recognition sequence is input into the projector and converted into target features aligned with the pre-trained large language model; The target features are decoded using the pre-trained large language model to generate a decision label for whether to accept the streaming recognition sequence. Specifically, when the target candidate text in the first candidate text set is consistent with the real labeled text, the pre-trained large language model outputs an acceptance decision flag; when the target candidate text in the first candidate text set is inconsistent with the real labeled text, the pre-trained large language model outputs a rejection decision flag. If the decision marker is the rejection decision marker, then the non-streaming recognition branch is activated to generate the second candidate text set.

[0012] In an exemplary embodiment of this disclosure, the projector includes a linear transform layer for feature downsampling processing, and the pre-trained large language model employs a multi-lexer parallel decoding architecture.

[0013] In an exemplary embodiment of this disclosure, the non-streaming recognition branch includes a non-streaming encoder and a connected temporal classification decoder; Activating the non-streaming recognition branch to generate the second candidate text set includes: The audio samples are encoded using the non-streaming encoder to generate non-streaming encoded features; The non-stream encoded features are decoded by the connected temporal classification decoder, and a second candidate text set containing multiple second candidate texts and their corresponding confidence scores is output.

[0014] In an exemplary embodiment of this disclosure, the step of iteratively training the initial large model based on the semantic difference between each second candidate text and the ground truth labeled text, and the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model, includes: A first loss value is determined based on the semantic difference between each second candidate text and the real labeled text; A second loss value is determined based on the text length difference between each second candidate text and the target candidate text; Determine the first target loss value based on the first loss value and the second loss value; The initial large model is iteratively trained based on the first target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

[0015] In an exemplary embodiment of this disclosure, determining a first loss value based on the semantic difference between each second candidate text and the real annotated text includes: Obtain the edit distance between each of the second candidate texts and the actual labeled text, and calculate a first average of the multiple edit distances; Calculate a first difference between each edit distance and the first average value, and obtain a first product between the first difference and the confidence score of each second candidate text; The first loss value is determined based on the sum of the multiple first products.

[0016] In an exemplary embodiment of this disclosure, determining the second loss value based on the text length difference between each second candidate text and the target candidate text includes: Obtain the specified length of the target candidate text; Obtain the length difference between each of the second candidate texts and the target candidate text, and calculate a second average of the multiple length differences; Calculate a second difference between each of the length differences and the second average value, and obtain a second product between the second difference and the confidence score of each of the second candidate texts; The second loss value is determined based on the sum of multiple second products.

[0017] In an exemplary embodiment of this disclosure, after determining the second loss value, the method further includes: The third loss value is determined based on the probability of the correct decision label being output by the decision branch; The second target loss value is determined based on the first loss value, the second loss value, and the third loss value; The initial large model is iteratively trained based on the second target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

[0018] In an exemplary embodiment of this disclosure, after obtaining the trained joint streaming and non-streaming speech recognition large model, the method further includes: Acquire the speech to be recognized; The speech to be recognized is input into the trained joint streaming and non-streaming speech recognition model, and a third candidate text set is generated through the streaming recognition branch. The credibility of the third candidate text set is evaluated through decision branches, and decision tags are generated. If the decision label is an acceptance-type decision label, then the final recognition result is determined based on the third candidate text set; If the decision marker is a rejection decision marker, then the non-streaming recognition branch is activated for re-recognition, and the final recognition result is output.

[0019] According to a second aspect of this disclosure, a training apparatus for a large speech recognition model is provided, comprising: The sample acquisition module is used to acquire a training dataset consisting of audio samples and their corresponding real-text annotations. The recognition processing module is used to input the training dataset into the initial large model and perform recognition processing on the audio samples through the streaming recognition branch to obtain the first candidate text set; The activation judgment module is used to determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text. The model training module is used to iteratively train the initial large model based on the semantic difference between each second candidate text and the real labeled text, as well as the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model.

[0020] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for a large speech recognition model as described in the first aspect.

[0021] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the training method for a large speech recognition model as described in the first aspect by executing the executable instructions.

[0022] As can be seen from the above technical solutions, the training method for the large-scale speech recognition model, the training device for the large-scale speech recognition model, the computer-readable storage medium, and the electronic device in the exemplary embodiments of this disclosure have at least the following advantages and positive effects: In some embodiments of this disclosure, a training dataset consisting of audio samples and their corresponding real-text annotations is obtained. This training dataset is then input into an initial large model, where a streaming recognition branch processes the audio samples to obtain a first candidate text set. Based on the text consistency between the target candidate text and the real-text annotations in the first candidate text set, it is determined whether to activate a non-streaming recognition branch to generate a second candidate text set. Based on the semantic differences between each second candidate text and the real-text annotations, and the text length differences between each second candidate text and the target candidate text, the initial large model is iteratively trained to obtain a well-trained model. The combined streaming and non-streaming speech recognition model achieves two key benefits. First, it optimizes the semantic accuracy and output stability of speech recognition results, effectively suppressing text length abrupt changes caused by non-streaming re-recognition while improving recognition accuracy, thus significantly alleviating the anticlimacticity problem in real-time captioning scenarios. Second, by introducing a text consistency-based conditional activation mechanism, it constructs an intelligent decision-making path driven by the large language model. This ensures that the model only triggers the high-cost non-streaming recognition branch when there is a significant deviation in the streaming recognition result. As a result, it reduces redundant computational overhead and system inference latency while ensuring overall recognition performance, thereby improving resource utilization efficiency and user experience.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 A flowchart illustrating the training method of a large speech recognition model in an embodiment of this disclosure is shown. Figure 2 This diagram illustrates how an audio sample is processed by a streaming recognition branch to obtain a first candidate text set in an embodiment of this disclosure. Figure 3 This illustration shows a flowchart of how, in an embodiment of the present disclosure, a non-streaming recognition branch is activated to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text. Figure 4 This diagram illustrates a flowchart of how a well-trained joint streaming and non-streaming speech recognition large model is obtained in the first embodiment of this disclosure. Figure 5 This diagram illustrates a second method for obtaining a trained joint streaming and non-streaming speech recognition large model in an embodiment of this disclosure. Figure 6 This diagram illustrates the architecture of the speech recognition system in an embodiment of this disclosure. Figure 7 This diagram illustrates the overall process of training a large speech recognition model according to an embodiment of the present disclosure. Figure 8 A schematic diagram illustrating how speech recognition is performed in an embodiment of this disclosure is shown; Figure 9 A schematic diagram of the structure of a training apparatus for a large speech recognition model in an exemplary embodiment of the present disclosure is shown. Figure 10 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0027] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0028] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0029] To improve the final accuracy of training large-scale speech recognition models, existing methods typically employ a two-pass recognition scheme. This involves accumulating a certain delay (e.g., determining the end of a sentence through voice activity detection) before performing a second pass of recognition using the complete speech information to update the original streaming recognition results. Existing training techniques for large-scale two-pass speech recognition models can be mainly divided into two categories: The first type of method is based on rescoring techniques: by training an additional scoring model (such as an attention decoding model), the Nbest candidate results generated by streaming recognition are rescored, and the candidate with the highest score (i.e., top1) is used as the final recognition result. The advantage of this method is that it does not require repeated calculation of audio samples, but the disadvantage is that the candidate results obtained based on streaming features may have poor accuracy, limiting the final two-pass effect.

[0030] The second type of method is based on non-streaming ASR technology. This involves using a non-streaming ASR model for secondary audio recognition to improve accuracy. However, current techniques primarily rely on traditional, low-parameter non-streaming models, leaving room for performance improvement. While large language models have achieved significant performance breakthroughs in training large speech recognition models, using them for secondary decoding still presents significant challenges. First, the high inference cost of large models significantly increases the overall hardware cost. Second, large models are slow inference. Since they typically employ autoregressive decoding, their latency is directly proportional to the length of the generated text. When the generated text is long, it may not meet real-time requirements. Finally, during the secondary decoding process of training a large speech recognition model, there may be significant differences in text length compared to the initial streaming decoding result, potentially causing the subtitles to jump on the screen and affecting readability.

[0031] In summary, although existing training techniques for large two-pass speech recognition models can effectively improve the final recognition accuracy, these techniques have shortcomings in performance, cost, and user experience, as detailed below: ① Limited accuracy improvement: The two-pass scheme based on rescoring or small model non-streaming ASR may have limited improvement in final accuracy due to the limitations of the accuracy of streaming candidate results or the recognition performance of small models. ② Large-scale two-pass inference has high cost and long latency: Although using a large-scale speech model can achieve better recognition results, the large number of model parameters leads to high inference cost. At the same time, the autoregressive decoding method often has a large latency. ③ Real-time display of animated subtitles: During the secondary decoding process, there may be a significant difference in text length between the initial decoding result and the streaming result, which may cause the subtitles to jump on the screen and affect readability.

[0032] In the embodiments of this disclosure, a training method for a large speech recognition model is first provided, which at least to some extent overcomes the shortcomings of related technologies that cannot alleviate the problem of subtitle jumping while improving recognition accuracy.

[0033] Figure 1 The diagram shows a flowchart of a method for training a large speech recognition model in an embodiment of this disclosure. The subject executing this method can be a server that trains the large speech recognition model.

[0034] refer to Figure 1 The training method for a large speech recognition model according to an embodiment of the present disclosure includes the following steps: Step S110: Obtain the training dataset consisting of audio samples and their corresponding real labeled text; Step S120: Input the training dataset into the initial large model, and use the streaming recognition branch to recognize and process the audio samples to obtain the first candidate text set; Step S130: Based on the text consistency between the target candidate text and the real labeled text in the first candidate text set, determine whether to activate the non-streaming recognition branch to generate the second candidate text set. Step S140: Based on the semantic difference between each second candidate text and the real labeled text, and the text length difference between each second candidate text and the target candidate text, the initial large model is iteratively trained to obtain a trained joint streaming and non-streaming speech recognition large model.

[0035] exist Figure 1 In the technical solution provided by the illustrated embodiment, a training dataset consisting of audio samples and their corresponding real-text annotations is obtained. This training dataset is then input into an initial large model, where the streaming recognition branch processes the audio samples to obtain a first candidate text set. Based on the text consistency between the target candidate text and the real-text annotations in the first candidate text set, it is determined whether to activate a non-streaming recognition branch to generate a second candidate text set. Based on the semantic differences between each second candidate text and the real-text annotations, as well as the text length differences between each second candidate text and the target candidate text, the initial large model is iteratively trained to obtain a well-trained joint... The large-scale streaming and non-streaming speech recognition model achieves joint optimization of semantic accuracy and output stability of speech recognition results. While improving recognition accuracy, it effectively suppresses abrupt changes in text length caused by non-streaming re-recognition, significantly alleviating the jumpiness problem in real-time subtitle scenarios. On the other hand, by introducing a conditional activation mechanism based on text consistency, it constructs an intelligent decision-making path driven by the large language model. This ensures that the model only triggers the high-cost non-streaming recognition branch when there is a significant deviation in the streaming recognition result. Thus, while ensuring overall recognition performance, it reduces redundant computational overhead and system inference latency, improving resource utilization efficiency and user experience.

[0036] The following are Figure 1 The specific implementation process of each step in the process will be explained in detail: In step S110, a training dataset consisting of audio samples and their corresponding real labeled text is obtained.

[0037] In this step, a training dataset consisting of audio samples and their corresponding real-text annotations can be obtained.

[0038] The training dataset mentioned above can be denoted as: ,in, Let i be the audio feature matrix corresponding to the i-th audio sample. is the corresponding real labeled text, and N is the number of samples in the training dataset mentioned above.

[0039] In an exemplary embodiment of this disclosure, the audio sample may be a 400,000-hour mixed Chinese and English speech sample set, and the audio sampling frequency may be 16 kHz; the audio feature matrix may be an 80-dimensional spectral feature, wherein the duration of each frame may be 25 ms, and the step size may be 10 ms. It should be noted that the above-mentioned values ​​can be flexibly set, and this disclosure does not impose any special limitations on them.

[0040] In step S120, the training dataset is input into the initial large model, and the audio samples are processed through the streaming recognition branch to obtain the first candidate text set.

[0041] In this step, the above training dataset can be input into the initial large model, and the audio samples can be processed through the streaming recognition branch to obtain the first candidate text set; each first candidate text is associated with a confidence score.

[0042] The initial large model can include a streaming recognition branch, a non-streaming recognition branch, and a decision branch. The streaming recognition branch includes a streaming encoder and a connection-temporal classification (CTC) decoder, used to encode and decode the input audio samples in real time, generating a first candidate text set containing multiple candidate texts and their corresponding confidence scores, which serves as the streaming recognition result. The decision branch receives the streaming recognition sequence constructed from this first candidate text set through a projector, and performs a credibility assessment based on its semantic distribution, confidence information, and consistency with the ground truth labeled text, outputting an acceptance class decision label (e.g., ...). <accept>) or rejection decision markers (such as <re-generation>When the output is an acceptance decision label, the final recognition result can be determined directly based on the target candidate text (usually the top-1 result) in the first candidate text set output by the streaming recognition branch. When the output is a rejection decision label, the non-streaming recognition branch is activated. This branch includes a non-streaming encoder and a CTC decoder. It uses the complete speech context to perform high-precision recognition again, generates a second candidate text set, and determines the final output result based on this set.

[0043] Given that the aforementioned streaming recognition branch includes a streaming encoder and a connected temporal classification decoder, therefore, referring to Figure 2 , Figure 2 This illustration shows a flowchart of how to obtain a first candidate text set by recognizing audio samples through a streaming recognition branch in an embodiment of this disclosure, including steps S201-S202: In step S201, the audio samples are encoded using a streaming encoder to generate streaming encoded features.

[0044] In this step, a streaming encoder can be used to encode the input audio samples segment by segment, based on time frames or speech blocks. For example, the streaming encoder described above can be implemented based on structures such as recurrent neural networks (RNNs), streaming variants of transformers (such as Triggered Attention, Conformer-Chunkwise Attention), or convolutional neural networks (CNNs). It can extract local acoustic features in real time and output the corresponding hidden state sequence, i.e., streaming encoded features, without relying on future context.

[0045] In step S202, the streaming encoded features are input to the connected temporal classification decoder to decode and obtain a set of first candidate texts containing multiple first candidate texts and their corresponding confidence scores.

[0046] In this step, the Connectionist Temporal Classification (CTC) decoder can be used to perform frame-by-frame alignment and decoding of the streaming encoded features generated in step S201. By introducing a blanking mechanism, the CTC decoder can automatically learn the mapping relationship between speech and text without forcibly aligning the input audio and output text, and output high-probability text and its corresponding score.

[0047] Specifically, the CTC decoder can calculate the output probability distribution of each word or sub-word unit at each time step based on the encoded features, and then generate multiple candidate recognition paths through a search strategy (such as greedy search, beam search, or multiple hypothesis generation). In this embodiment, a beam search strategy can be used to retain several paths with the highest scores, thereby forming a first candidate text set containing multiple first candidate texts and their corresponding confidence scores.

[0048] For example, during the decoding process of the current speech segment, the top-3 candidate results generated could be: "Today is Sunday," with a confidence score of 0.8; "Today is a day of mood," with a confidence score of 0.2; and "Today is a day of clarity," with a confidence score of 0.05. This set not only includes the optimal recognition hypothesis (such as the top-1), but also retains the second-best candidates that are semantically similar or phonetically similar (such as "a day of mood" and "Sunday" being easily confused phonetically), providing rich contextual information and error correction basis for subsequent decision-making, semantic consistency analysis, or non-streaming re-recognition.

[0049] Furthermore, the aforementioned confidence scores can be normalized to relative probabilities or retained as log-likelihood forms for use in subsequent modules such as fusion scoring, decision threshold determination, or candidate sequence ranking. By generating multiple candidate results, this step effectively improves the system's ability to model uncertainty in identification and enhances overall identification robustness.

[0050] Next, refer to Figure 1 In step S130, based on the text consistency between the target candidate text and the real labeled text in the first candidate text set, it is determined whether to activate the non-streaming recognition branch to generate the second candidate text set.

[0051] In this step, the initial large model may also include decision branches, which may include a frozen pre-trained large language model (LLM) and two trainable lightweight modules: Projector: Used to map input sequences to the input embedding space of a pre-trained large language model, achieving modality alignment; LoRA module: Embedded in the attention layer inside the pre-trained large language model, it is used to introduce a small number of trainable parameters to adapt to downstream decision-making tasks.

[0052] During training, the backbone parameters of the pre-trained large language model are kept frozen, and only the Projector and LoRA modules are optimized, thereby giving the model semantic decision-making capabilities while controlling computational costs.

[0053] The aforementioned decision branch can determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text and the ground truth labeled text in the first candidate text set. For details, refer to... Figure 3 , Figure 3 This embodiment of the present disclosure illustrates how to determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text, including steps S301-S303: In step S301, multiple first candidate texts and their confidence scores are concatenated according to preset rules to form a streaming recognition sequence.

[0054] In this step, the input sequence can be constructed using a structured concatenation method based on the number of candidates in the first candidate text set (e.g., N-best) and their corresponding confidence scores. Specifically, the top-k candidate texts can be sorted in descending order of their scores, and their text content can be combined with the normalized scores to form a formatted string sequence. For example, a streaming recognition sequence of the following form can be constructed: "Candidate 1: Today is Sunday (score=0.8); Candidate 2: Today's mood is good (score=0.2); Candidate 3: Today is clear (score=0.05)". This sequence serves as a context-rich prompt, explicitly expressing the output distribution of the streaming recognition model, the relative confidence between candidates, and potential semantic ambiguities, providing a basis for judgment for subsequent large language models, enabling them to assess the reliability of the current recognition results.

[0055] In step S302, the streaming recognition sequence is input to the projector and converted into target features aligned with the pre-trained large language model.

[0056] In this step, the above-mentioned streaming recognition sequence can be input into the projector for feature alignment processing to convert it into target features aligned with the pre-trained large language model.

[0057] In step S303, the target features are decoded using a pre-trained large language model to generate a decision label for whether to accept the streaming recognition sequence.

[0058] In this step, the pre-trained large language model can perform a binary decision classification task: output <accept>or <re-generation>A flag is added to indicate whether the current streaming identification result should be adopted as the final output.

[0059] During the training phase, this task employs supervised learning. The supervised labels are automatically constructed based on the textual consistency between the target candidate texts (usually the top-1 results) in the first candidate text set and the corresponding ground truth labeled texts in the audio. If the top-1 results are completely consistent with the ground truth labeled texts, then the expected output is constructed as the acceptance class decision label. <accept>If the top-1 result differs from the actual labeled text (e.g., "Today's mood is sunny" ≠ "Today is Sunday"), then construct the expected output as a rejection class decision label. <re-generation>.

[0060] For example, when the top-1 score is significantly higher than other candidates and the semantics are reasonable, the model tends to generate... <accept>When the top-1 score is close to other candidate scores, there is semantic confusion, or the context is unreasonable, the model tends to generate... <re-generation>.

[0061] Ultimately, if the generated decision label is <re-generation>If the streaming recognition branch is activated, high-precision recognition is performed again using the complete speech context to generate a second candidate text set as the basis for the final recognition result. Specifically, the streaming recognition branch includes a non-streaming encoder and a connection-temporal classification (CTC) decoder. The process of activating the streaming recognition branch to generate the second candidate text set may include the following steps: forward encoding the audio samples (or accumulated complete speech segments) using the non-streaming encoder to fully model the global context information and generate high-fidelity non-streaming encoded features; inputting the non-streaming encoded features into the connection-temporal classification decoder for decoding based on acoustic-text alignment capability, and outputting a second candidate text set containing multiple candidate texts and their corresponding confidence scores. This second candidate text set has a higher accuracy rate than the streaming recognition result, especially when dealing with easily confused words, low signal-to-noise ratio speech, or complex contexts. Therefore, this disclosure can select the best candidate (such as the top-1 result) as the final recognition output, or further re-scoring and ranking to improve the overall recognition quality.

[0062] It should be noted that the projector in this disclosure includes a linear transformation layer for feature downsampling. Specifically, this linear layer can downsample the encoded features of the input streaming recognition sequence by a factor of 2 in the time dimension to compress the sequence length, reduce the computational load, and map it to the input embedding space of the large language model. This design not only reduces the length of the input sequence of the subsequent large language model, thereby improving the overall inference efficiency, but also preserves key semantic information to ensure the accuracy of decision-making.

[0063] Furthermore, pre-trained large language models can employ a multi-token prediction (MTP) architecture. In each decoding step, the model can simultaneously predict and output multiple consecutive output tokens, rather than the token-by-token generation method of traditional autoregressive models. This architecture significantly improves the efficiency of decision token generation, and is particularly suitable for... <accept>or <re-generation>Structured, fixed-pattern output tasks. For example, after receiving downsampled target features, a pre-trained large language model can directly output complete decision tags (such as complete token sequences) in a single forward computation. <accept>Instead of generating characters one by one, this effectively reduces the number of decoding steps and lowers end-to-end latency. Combining the Projector's downsampling capabilities with the parallel output characteristics of the MTP architecture, this solution can achieve highly efficient streaming recognition result credibility assessment while ensuring decision accuracy.

[0064] Next, refer to Figure 1 In step S140, the initial large model is iteratively trained based on the semantic difference between each second candidate text and the real labeled text, as well as the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model.

[0065] In this step, the initial large model can be iteratively trained based on the semantic differences between each second candidate text and the real labeled text, as well as the text length differences between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model.

[0066] In one exemplary implementation, reference Figure 4 , Figure 4 This diagram illustrates a first method for obtaining a trained joint streaming and non-streaming speech recognition large model according to an embodiment of the present disclosure, including steps S401-S404: In step S401, a first loss value is determined based on the semantic difference between each second candidate text and the real labeled text.

[0067] In this step, we assume that the Nbest candidate in the non-streaming identification results is , Each second candidate text (i takes values ​​from 1 to N) represents a different text, thus allowing us to determine the text based on each candidate text. The semantic difference between the text and the actual labeled text y is used to determine the first loss value.

[0068] Specifically, the first loss value mentioned above can be calculated based on the following formula 1. :

[0069] Specifically, each second candidate text can be obtained. Edit distance between the actual annotated text y And calculate the first average of multiple edit distances. ; Calculate the first difference between each edit distance and the first average value. And obtain the first product between the first difference and the confidence score of each second candidate text. Next, the first loss value can be determined based on the sum of the multiple first products.

[0070] In step S402, a second loss value is determined based on the text length difference between each second candidate text and the target candidate text.

[0071] In this step, the second loss value can be determined based on the text length difference between each second candidate text and the target candidate text.

[0072] Specifically, the second loss value mentioned above can be determined based on the following formula 2. :

[0073] Specifically, it can obtain the target candidate text from the first candidate text set output by the streaming recognition branch. The specified length is then used to obtain each second candidate text. With the above target candidate text The difference in length between And calculate the second average of multiple length differences. Next, the second difference between each length difference and the second average can be calculated. And obtain the second product between the second difference and the confidence score of each second candidate text. Finally, the second loss value is determined based on the sum of multiple second products.

[0074] In step S403, the first target loss value is determined based on the first loss value and the second loss value.

[0075] In this step, the first target loss value can be determined based on the first loss value and the second loss value mentioned above. For example, the first loss value and the second loss value can be summed to determine the first target loss value, or the first loss value and the second loss value can be weighted to determine the first target loss value. Both can be set according to the actual situation, and this disclosure does not impose any special limitations on them.

[0076] In step S404, the initial large model is iteratively trained based on the first target loss value to obtain a trained joint streaming and non-streaming speech recognition large model.

[0077] In this step, the initial large model can be iteratively trained based on the first target loss value mentioned above, and the trainable parameters in the model can be continuously adjusted, including the streaming encoder, non-streaming encoder, connected temporal classifier decoder, projector network, and LoRA module. The parameters are updated through backpropagation to gradually reduce the overall loss until the first target loss value converges or reaches the preset number of training rounds, and finally a converged and stable trained large model of joint streaming and non-streaming speech recognition is obtained.

[0078] In an alternative implementation, after step S403 described above, reference can be made to... Figure 5 , Figure 5 This diagram illustrates a second method for obtaining a trained joint streaming and non-streaming speech recognition large model in this embodiment of the present disclosure, including steps S501-S503: In step S501, the third loss value is determined based on the probability of the decision branch outputting the correct decision label.

[0079] In this step, the third loss value The third loss value is used to measure the predictive accuracy of the decision branch on the decision task, which can be achieved by calculating the difference between the model output distribution and the expected decision label. Preferably, the third loss value adopts the cross-entropy loss to effectively supervise the model to learn when to accept the streaming recognition result and when to reject it and trigger non-streaming re-recognition.

[0080] Specifically, during training, a binary supervised label can be constructed based on the consistency between the target candidate texts (usually the top-1 results) in the first candidate text set and the corresponding ground truth labeled text of the audio: if the two are consistent, the expected output is the acceptor decision label. <accept>If they are inconsistent, the expected output is a rejection decision label. <re-generation>The decision branch receives the streaming recognition sequence mapped by the Projector and generates a probability distribution for the aforementioned decision labels. The third loss value is the cross-entropy between this probability distribution and the true decision labels.

[0081] By minimizing the third loss value, the decision branch can be guided to learn to capture risk signals of identification errors from the semantic distribution, confidence differences, and contextual consistency of the N-best candidate texts, thereby improving its decision reliability in real-world scenarios. For example, when the top-1 result has high confidence and reasonable semantics, the model is encouraged to output a high probability. <accept>However, when multiple competing hypotheses or semantic contradictions exist, the output is guided. <re-generation>.

[0082] In step S502, a second target loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0083] In this step, the second target loss value can be determined based on the following formula 3. :

[0084] in, and The hyperparameters are used to train the system and balance the effects of different losses. The specific values ​​can be set according to the actual situation, and this disclosure does not impose any special restrictions on them.

[0085] In step S503, the initial large model is iteratively trained according to the second target loss value to obtain a trained joint streaming and non-streaming speech recognition large model.

[0086] In this step, end-to-end parameter optimization can be performed on the trainable components of the initial large model based on the aforementioned second objective loss value. Specifically, the loss gradient can be propagated to each module of the model through backpropagation, including the streaming encoder, non-streaming encoder, connected temporal classification decoder, projector network, and LoRA module. Adaptive optimization algorithms (such as AdamW) are then used to update the parameters, gradually reducing the overall loss. During training, batch iterative processing of training samples is employed, combined with strategies such as learning rate scheduling and gradient pruning to improve training stability and convergence speed. Through multiple iterations, the model learns to control the text length variation of the non-streaming re-recognition output while ensuring semantic accuracy, thereby improving the stability of the recognition results, as it minimizes the second objective loss value. When the loss value tends to converge or reaches the preset number of training rounds, model training is completed, resulting in the final trained joint streaming and non-streaming speech recognition large model.

[0087] refer to Figure 6 , Figure 6 This diagram illustrates the architecture of the speech recognition system in an embodiment of this disclosure, such as... Figure 6 As shown: In the speech recognition system of this disclosure embodiment, a dual-mode recognition technology combining streaming and non-streaming methods is used to improve the accuracy and efficiency of recognition.

[0088] Audio encoding: First, the input audio samples are processed in real time using a streaming encoder to generate streaming encoded features. These features capture key information in the audio, providing a foundation for subsequent decoding; CTC Decoding and Initial Candidate Text Set Generation: Next, the streaming coding features are decoded using the CTC decoder to generate a series of first candidate texts and their corresponding confidence scores. For example, the confidence score for "Today is Sunday" is 0.8, "Today is a day of feeling" is 0.2, and "Today is a day of clarity" is 0.05, forming the first candidate text set. Decision Guidance and Projection: Subsequently, the first candidate text set obtained from the CTC decoder is fed into a projector module containing a linear transformation layer. This module is responsible for mapping the streaming recognition results to the input space of a pre-trained Large Language Model (LLM), where the LLM, as a core component, employs a pre-trained frozen model and embeds a LoRA (Low-Rank Adaptation) module to guide the decision-making process. By analyzing input features, the LLM can evaluate the reliability of the current streaming recognition results and output an acceptance or rejection decision label; Final recognition result output: If the LLM output is an acceptance marker, the first candidate text with the highest confidence is directly output as the final recognition result; otherwise, if it is a rejection marker, the non-streaming recognition branch is activated, and the complete context of the entire audio segment is used for more accurate recognition, thereby generating a second candidate text set to ensure the accuracy of the final recognition result.

[0089] In summary, this process, through multi-level information fusion and decision optimization, achieves efficient and reliable speech-to-text functionality, significantly improving recognition accuracy and user experience.

[0090] refer to Figure 7 , Figure 7 This diagram illustrates the overall flowchart of the training method for a large speech recognition model in this embodiment, including steps S701-S705: In step S701, training data is prepared; a training dataset consisting of "speech" and "text" pairs is constructed, and audio features are extracted; In step S702, the streaming / non-streaming speech recognition branches are trained jointly. Specifically, the open-source speech toolkit wenet is used to implement a unified encoder-decoder structure based on the Conformer architecture, where the encoder shares the feature extraction capabilities of both streaming and non-streaming paths. The model adopts a joint training objective of CTC + Attention, balancing alignment robustness and semantic modeling capabilities. By introducing a dynamic feature window mechanism, the same model can support streaming (low latency) and non-streaming (high accuracy) recognition under different inference modes. Finally, the CTC decoding output is used as the streaming recognition result to obtain a small ASR model (approximately 200M parameters) with dual-mode recognition capabilities. In step S703, large-scale multimodal alignment training is performed. Specifically, a multimodal architecture is constructed, comprising a streaming or non-streaming recognition branch, a modality alignment module (Projector), and a pre-trained large language model (LLM). The Projector module includes a linear transformation layer, used to map the encoded features or cue vectors corresponding to the N-best candidate text sequences generated by streaming recognition to the input embedding space of the LLM, achieving semantic alignment between speech and text modalities. In this stage, the LLM backbone parameters are fixed, and only the Projector is trained, enabling it to effectively inject ASR output into the large-scale model context, providing support for subsequent decision-making. In step S704, the large model generation controls the training. Specifically, for each training sample, the pre-trained streaming ASR model first generates an N-best recognition result and its confidence score. Then, based on the semantic consistency between the top-1 recognition results and the real text annotations, a supervision signal is constructed: if consistent, the large language model is expected to output... <accept>Mark; if inconsistent, the expected output is... <re-generation>mark; In step S705, the alignment error rate is minimized. Specifically, based on the difference in length between the non-streaming Nbest candidates and the top 1 output of the streaming ASR model, the large model is encouraged to generate candidates with similar lengths to the streaming results but more accurate. The training converges to obtain the final large speech recognition model.

[0091] After obtaining a well-trained speech recognition model, the model inference stage can begin. For details, please refer to... Figure 8 , Figure 8 This diagram illustrates a flowchart of how speech recognition is performed in an embodiment of this disclosure, including steps S801-S805: In step S801, the speech to be recognized is acquired.

[0092] In this step, raw audio signals from user input or real-time acquisition can be obtained as the speech data to be recognized. This audio can be single-sentence speech, continuous dialogue, or conference recordings, and is suitable for various application scenarios such as conference transcription, real-time captioning, and voice assistants.

[0093] In step S802, the large joint streaming and non-streaming speech recognition model trained on the speech input to be recognized is used to generate a third candidate text set through the streaming recognition branch.

[0094] In this step, after the speech to be recognized is input into the pre-trained joint streaming and non-streaming speech recognition model, the model first activates the streaming recognition branch (including the streaming encoder and CTC decoder) to perform low-latency online recognition of the input speech. This branch outputs a third candidate text set in real time, containing multiple candidate results and their corresponding confidence scores, such as "Today is Sunday" (score=0.8), "Today's mood is..." (score=0.2), etc. This set reflects the model's initial judgment and uncertainty distribution under the current acoustic conditions, providing a basis for subsequent confidence assessment.

[0095] In step S803, the credibility of the third candidate text set is evaluated through decision branches, and decision tags are generated.

[0096] In this step, the third candidate text set can be concatenated according to a preset format (such as a structured prompt) and mapped by a projector to a feature representation aligned with the input space of the large language model. Subsequently, the frozen backbone of the large language model, combined with its internal LoRA adaptation module, decodes this representation, comprehensively analyzes the semantic rationality, confidence distribution, and contextual consistency of the candidate texts, and outputs a structured decision label. <accept>or <re-generation>This is used to indicate whether to accept the current streaming identification result.

[0097] In step S804, if the decision label is an acceptance-type decision label, the final recognition result is determined based on the third candidate text set.

[0098] In this step, when the decision branch outputs... <accept>When the confidence level is high enough, it indicates that the streaming recognition result is reliable and no further correction is needed. At this point, the candidate with the highest confidence level (usually top-1) can be directly selected from the third candidate text set as the final recognition result output, achieving low latency and high efficiency speech transcription.

[0099] In step S805, if the decision label is a rejection decision label, the non-streaming recognition branch is activated for re-recognition, and the final recognition result is output.

[0100] In this step, when the decision label is <re-generation>When the streaming recognition result has a high risk of error, the non-streaming recognition branch (including the non-streaming encoder and CTC decoder) is activated. This branch utilizes complete speech context information to perform a high-precision re-recognition, generating a more accurate set of candidate texts. Finally, the optimal result is selected as the output, significantly improving recognition accuracy, especially in cases of easily confused words, low signal-to-noise ratios, or complex contexts.

[0101] Based on the above technical solutions, this disclosure has at least the following technical effects: First, it improves the accuracy and reliability of speech recognition decisions. By introducing decision branches into a large-scale joint streaming and non-streaming speech recognition model, and combining the N-best candidate texts and their confidence scores from the streaming recognition results, semantic-level credibility assessment of the recognition results is performed, and a model is generated. <accept>or <re-generation>Decision labeling enables intelligent judgment and selective error correction of streaming recognition errors. Especially when facing complex scenarios such as ambiguous pronunciation, background noise, or easily confused words, this mechanism can effectively identify high-risk recognition results and trigger non-streaming re-recognition only when necessary. This ensures overall recognition accuracy while improving the system's semantic understanding ability and decision robustness.

[0102] Second, it reduces the inference overhead and system latency of large models. Through generative control training, the decision branches acquire dynamic decision-making capabilities: when the streaming recognition results are consistent with the true semantics, the output... <accept>Decoding is prematurely terminated by marking the input tokens to avoid unnecessary computation; a non-streaming recognition branch is triggered for high-precision re-recognition only when the recognition result shows a significant deviation. Simultaneously, the pre-trained large language model employs a multi-token prediction architecture, supporting single-step generation of complete decision tokens and significantly reducing the number of autoregressive decoding steps. Combined with the feature downsampling mechanism in the projector, the input sequence length is further shortened, reducing computational complexity. The overall solution effectively reduces the computational load on large models and shortens end-to-end response time while maintaining recognition quality, making it suitable for latency-sensitive and resource-constrained real-time speech recognition scenarios.

[0103] Third, it alleviates the text jump problem in real-time caption display, improving output stability and user experience. By introducing constraints on the text length difference between the second candidate text sequence and the target candidate text during training, and combining this with semantic accuracy targets for joint optimization, the non-streaming re-recognition results maintain length consistency with the original streaming output as much as possible while improving semantic correctness. This mechanism effectively suppresses visual jump phenomena such as caption scaling, line breaks, or positional shifts caused by secondary recognition, significantly improving the user reading experience in scenarios with high requirements for display continuity, such as conference transcription and live captions.

[0104] Fourth, it enhances the adaptability and training efficiency of large language models in speech context. By constructing structured prompts from streaming recognition results and aligning them to the input space of the large language model via a projector, it provides the large model with rich acoustic-semantic contextual information, enhancing its ability to understand and model speech recognition error patterns. This conditional input mechanism not only improves the performance of the large model in error correction tasks but also promotes the training convergence speed and decoding accuracy of multi-lexical prediction structures, achieving effective collaboration between speech and language modalities.

[0105] This disclosure also provides a training apparatus for a large speech recognition model. Figure 9 This diagram illustrates the structure of a training apparatus for a large speech recognition model in an exemplary embodiment of this disclosure; as shown... Figure 9 As shown, the training device 900 for a large speech recognition model may include a sample acquisition module 910, a recognition processing module 920, an activation judgment module 930, and a model training module 940. Wherein: The sample acquisition module 910 is used to acquire a training dataset consisting of audio samples and their corresponding real labeled text. The recognition processing module 920 is used to input the training dataset into the initial large model and perform recognition processing on the audio samples through the streaming recognition branch to obtain the first candidate text set; The activation judgment module 930 is used to determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text. The model training module 940 is used to iteratively train the initial large model based on the semantic difference between each second candidate text and the real labeled text and the text length difference between each second candidate text and the target candidate text, so as to obtain a trained joint streaming and non-streaming speech recognition large model.

[0106] In an exemplary embodiment of this disclosure, the streaming identification branch includes a streaming encoder and a connected temporal classification decoder; The recognition processing module 920 performs recognition processing on the audio samples through its streaming recognition branch to obtain a first candidate text set, including: The audio samples are encoded using the streaming encoder to generate streaming encoded features; The streaming encoding features are input into the connection-time classification decoder to decode and obtain a first candidate text set containing multiple first candidate texts and their corresponding confidence scores.

[0107] In an exemplary embodiment of this disclosure, the initial large model further includes a decision branch; the decision branch includes a projector and a pre-trained large language model; The activation judgment module 930 determines whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text, including: Multiple first candidate texts and their confidence scores are concatenated according to preset rules to form a streaming recognition sequence; The streaming recognition sequence is input into the projector and converted into target features aligned with the pre-trained large language model; The target features are decoded using the pre-trained large language model to generate a decision label for whether to accept the streaming recognition sequence. Specifically, when the target candidate text in the first candidate text set is consistent with the real labeled text, the pre-trained large language model outputs an acceptance decision flag; when the target candidate text in the first candidate text set is inconsistent with the real labeled text, the pre-trained large language model outputs a rejection decision flag. If the decision marker is the rejection decision marker, then the non-streaming recognition branch is activated to generate the second candidate text set.

[0108] In an exemplary embodiment of this disclosure, the projector includes a linear transform layer for feature downsampling processing, and the pre-trained large language model employs a multi-lexer parallel decoding architecture.

[0109] In an exemplary embodiment of this disclosure, the non-streaming recognition branch includes a non-streaming encoder and a connected temporal classification decoder; The activation judgment module 930 activates the non-streaming recognition branch to generate the second candidate text set, including: The audio samples are encoded using the non-streaming encoder to generate non-streaming encoded features; The non-stream encoded features are decoded by the connected temporal classification decoder, and a second candidate text set containing multiple second candidate texts and their corresponding confidence scores is output.

[0110] In an exemplary embodiment of this disclosure, the model training module 940 iteratively trains the initial large model based on the semantic difference between each second candidate text and the ground truth labeled text, and the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model, including: A first loss value is determined based on the semantic difference between each second candidate text and the real labeled text; A second loss value is determined based on the text length difference between each second candidate text and the target candidate text; Determine the first target loss value based on the first loss value and the second loss value; The initial large model is iteratively trained based on the first target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

[0111] In an exemplary embodiment of this disclosure, the model training module 940 determines a first loss value based on the semantic difference between each second candidate text and the ground truth labeled text, including: Obtain the edit distance between each of the second candidate texts and the actual labeled text, and calculate a first average of the multiple edit distances; Calculate a first difference between each edit distance and the first average value, and obtain a first product between the first difference and the confidence score of each second candidate text; The first loss value is determined based on the sum of the multiple first products.

[0112] In an exemplary embodiment of this disclosure, the model training module 940 determines a second loss value based on the text length difference between each second candidate text and the target candidate text, including: Obtain the specified length of the target candidate text; Obtain the length difference between each of the second candidate texts and the target candidate text, and calculate a second average of the multiple length differences; Calculate a second difference between each of the length differences and the second average value, and obtain a second product between the second difference and the confidence score of each of the second candidate texts; The second loss value is determined based on the sum of multiple second products.

[0113] In an exemplary embodiment of this disclosure, after determining the second loss value, the model training module 940 is configured to: The third loss value is determined based on the probability of the correct decision label being output by the decision branch; The second target loss value is determined based on the first loss value, the second loss value, and the third loss value; The initial large model is iteratively trained based on the second target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

[0114] In an exemplary embodiment of this disclosure, after obtaining the trained joint streaming and non-streaming speech recognition large model, the model inference module (not shown in the figure) is configured as follows: Acquire the speech to be recognized; The speech to be recognized is input into the trained joint streaming and non-streaming speech recognition model, and a third candidate text set is generated through the streaming recognition branch. The credibility of the third candidate text set is evaluated through decision branches, and decision tags are generated. If the decision label is an acceptance-type decision label, then the final recognition result is determined based on the third candidate text set; If the decision marker is a rejection decision marker, then the non-streaming recognition branch is activated for re-recognition, and the final recognition result is output.

[0115] The specific details of each module in the training device of the aforementioned large speech recognition model have been described in detail in the training method of the corresponding large speech recognition model, so they will not be repeated here.

[0116] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0117] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0118] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0119] This disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0120] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0121] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0122] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0123] Furthermore, this disclosure also provides an electronic device capable of implementing the above-described method.

[0124] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0125] The following reference Figure 10 To describe an electronic device 1000 according to such an embodiment of the present disclosure. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0126] like Figure 10 As shown, the electronic device 1000 is presented in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processor 1010, at least one memory 1020, a bus 1030 connecting different system components (including memory 1020 and processor 1010), and a display 1040.

[0127] The memory stores program code that can be executed by the processor 1010, causing the processor 1010 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure. For example, the processor 1010 can perform actions such as... Figure 1 As shown: Step S110, obtain a training dataset consisting of audio samples and their corresponding real labeled text; Step S120, input the training dataset into the initial large model, and use the streaming recognition branch to recognize the audio samples to obtain a first candidate text set; Step S130, determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text and the real labeled text in the first candidate text set; Step S140, iteratively train the initial large model based on the semantic difference between each second candidate text and the real labeled text and the text length difference between each second candidate text and the target candidate text to obtain a trained joint streaming and non-streaming speech recognition large model.

[0128] The memory 1020 may include a readable medium in the form of volatile storage, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.

[0129] The memory 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0130] Bus 1030 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0131] Electronic device 1000 can also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.< / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept> < / accept>

Claims

1. A training method for a large-scale speech recognition model, characterized in that, include: Obtain a training dataset consisting of audio samples and their corresponding real-text annotations; The training dataset is input into the initial large model, and the audio samples are processed through the streaming recognition branch to obtain the first candidate text set; Based on the text consistency between the target candidate text in the first candidate text set and the real labeled text, determine whether to activate the non-streaming recognition branch to generate the second candidate text set. Based on the semantic difference between each second candidate text and the real labeled text, and the text length difference between each second candidate text and the target candidate text, the initial large model is iteratively trained to obtain a trained joint streaming and non-streaming speech recognition large model.

2. The method according to claim 1, characterized in that, The streaming recognition branch includes a streaming encoder and a connected temporal classification decoder; The step of obtaining a first candidate text set by recognizing the audio samples through the streaming recognition branch includes: The audio samples are encoded using the streaming encoder to generate streaming encoded features; The streaming encoding features are input into the connection-time classification decoder to decode and obtain a first candidate text set containing multiple first candidate texts and their corresponding confidence scores.

3. The method according to claim 1, characterized in that, The initial large model also includes decision branches; the decision branches include projectors and pre-trained large language models. The step of determining whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text includes: Multiple first candidate texts and their confidence scores are concatenated according to preset rules to form a streaming recognition sequence; The streaming recognition sequence is input into the projector and converted into target features aligned with the pre-trained large language model; The target features are decoded using the pre-trained large language model to generate a decision label for whether to accept the streaming recognition sequence. Specifically, when the target candidate text in the first candidate text set is consistent with the real labeled text, the pre-trained large language model outputs an acceptance decision flag; when the target candidate text in the first candidate text set is inconsistent with the real labeled text, the pre-trained large language model outputs a rejection decision flag. If the decision marker is the rejection decision marker, then the non-streaming recognition branch is activated to generate the second candidate text set.

4. The method according to claim 3, characterized in that, The projector includes a linear transformation layer for feature downsampling, and the pre-trained large language model employs a multi-lexer parallel decoding architecture.

5. The method according to claim 3, characterized in that, The non-streaming recognition branch includes a non-streaming encoder and a connected temporal classification decoder; Activating the non-streaming recognition branch to generate the second candidate text set includes: The audio samples are encoded using the non-streaming encoder to generate non-streaming encoded features; The non-stream encoded features are decoded by the connected temporal classification decoder, and a second candidate text set containing multiple second candidate texts and their corresponding confidence scores is output.

6. The method according to claim 1, characterized in that, The step of iteratively training the initial large model based on the semantic difference between each second candidate text and the ground truth labeled text, and the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model, includes: A first loss value is determined based on the semantic difference between each second candidate text and the real labeled text; A second loss value is determined based on the text length difference between each second candidate text and the target candidate text; Determine the first target loss value based on the first loss value and the second loss value; The initial large model is iteratively trained based on the first target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

7. The method according to claim 6, characterized in that, The step of determining the first loss value based on the semantic difference between each second candidate text and the ground truth labeled text includes: Obtain the edit distance between each of the second candidate texts and the actual labeled text, and calculate a first average of the multiple edit distances; Calculate a first difference between each edit distance and the first average value, and obtain a first product between the first difference and the confidence score of each second candidate text; The first loss value is determined based on the sum of the multiple first products.

8. The method according to claim 6, characterized in that, The step of determining the second loss value based on the text length difference between each second candidate text and the target candidate text includes: Obtain the specified length of the target candidate text; Obtain the length difference between each of the second candidate texts and the target candidate text, and calculate a second average of the multiple length differences; Calculate a second difference between each of the length differences and the second average value, and obtain a second product between the second difference and the confidence score of each of the second candidate texts; The second loss value is determined based on the sum of multiple second products.

9. The method according to claim 6, characterized in that, After determining the second loss value, the method further includes: The third loss value is determined based on the probability of the correct decision label being output by the decision branch; The second target loss value is determined based on the first loss value, the second loss value, and the third loss value; The initial large model is iteratively trained based on the second target loss value to obtain the trained joint streaming and non-streaming speech recognition large model.

10. The method according to any one of claims 1 to 9, characterized in that, After obtaining the trained joint streaming and non-streaming speech recognition large model, the method further includes: Acquire the speech to be recognized; The speech to be recognized is input into the trained joint streaming and non-streaming speech recognition model, and a third candidate text set is generated through the streaming recognition branch. The credibility of the third candidate text set is evaluated through decision branches, and decision tags are generated. If the decision label is an acceptance-type decision label, then the final recognition result is determined based on the third candidate text set; If the decision marker is a rejection decision marker, then the non-streaming recognition branch is activated for re-recognition, and the final recognition result is output.

11. A training device for a large-scale speech recognition model, characterized in that, include: The sample acquisition module is used to acquire a training dataset consisting of audio samples and their corresponding real-text annotations. The recognition processing module is used to input the training dataset into the initial large model and perform recognition processing on the audio samples through the streaming recognition branch to obtain the first candidate text set; The activation judgment module is used to determine whether to activate the non-streaming recognition branch to generate a second candidate text set based on the text consistency between the target candidate text in the first candidate text set and the real labeled text. The model training module is used to iteratively train the initial large model based on the semantic difference between each second candidate text and the real labeled text, as well as the text length difference between each second candidate text and the target candidate text, to obtain a trained joint streaming and non-streaming speech recognition large model.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for the large speech recognition model according to any one of claims 1 to 10.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the training method for a large speech recognition model according to any one of claims 1 to 10 by executing the executable instructions.

Images