A hybrid speech recognition method and system based on an AI large model architecture

Abstract

The application provides a hybrid speech recognition method and system based on an AI large model architecture, and the method comprises the following steps: a hybrid system of an ASR system containing a traditional ASR system and an AI large model is constructed; based on a large amount of unlabeled audio data, the large model is pre-trained through self-supervised learning, the audio signal features are mined, and the AI large model is supervised fine-tuned; the text output is trained using the cross-entropy loss in the basic layer; the acoustic modeling branch is introduced in the middle layer or the auxiliary head, the acoustic output is trained using the CTC loss function; the acoustic and text information fusion is realized by training the CTC loss function and the cross-entropy loss; in the model training or inference stage, the audio blocking mode and the KV cache technology are used to realize efficient streaming recognition; the recognition result is supported while speaking; after the speech is finished, the large model quickly completes the subsequent autoregressive text inference based on all the cached information, and outputs high-precision recognition text. The application realizes the compatibility of high precision and streaming experience.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, speech recognition, and natural language processing. In particular, it relates to a hybrid speech recognition technology that combines a large AI model with a traditional automatic speech recognition (ASR) system architecture; more specifically, it relates to a hybrid speech recognition method and system based on a large AI model architecture. Background Technology

[0002] With the rapid popularization of voice interaction in mobile internet, smart terminals, and office scenarios, ASR (Audio Recognition) technology has become the foundation of audio understanding. Traditional ASR systems use a separate modeling approach for acoustic and language models, with a system architecture such as... Figure 1 As shown, acoustic models require supervised training with a large amount of labeled data, while language models typically employ N-gram techniques, requiring only text data. The acoustic and language models are fused by a decoder to output the final recognition result. This architecture facilitates the use of domain-specific text to augment the language model, achieving rapid adaptation to specific scenarios and reducing customization costs. However, in traditional ASR systems, the acoustic and language models are trained independently, leading to insufficient information interaction, error accumulation, and difficulty in fully utilizing massive amounts of unlabeled audio resources, significantly limiting the improvement of recognition performance.

[0003] In recent years, with the development of Large Language Model (LLM) technology, end-to-end ASR solutions have gradually emerged. For example, a large text AI model can be extended into a speech model by adapting the input layer (such as a speech segmenter), utilizing encoder-decoder or decoder-only architectures, and employing self-supervised learning methods and training with massive amounts of speech or text data. The architecture diagram of an ASR (speech recognition) system based on Large Language Model (LLM) is shown below. Figure 2 As shown, due to the large-scale increase in both the amount of training data and the number of model parameters, the LLM-based ASR system has achieved a significant improvement in overall recognition accuracy and recognition rate in noisy environments, and has extremely strong scalability in terms of parameters and data scale.

[0004] However, a drawback of LLM-based ASR systems is that autoregressive decoding is non-stream recognition, such as... Figure 3As shown, autoregressive decoding can only begin after the user finishes speaking, further increasing the latency of human-computer interaction. Furthermore, it cannot implement streaming display functionality [streaming display refers to the real-time display of recognition results on the screen during voice interaction on devices with screens (such as in-vehicle systems, TVs, and smart speakers with screens)]. Therefore, it is unsuitable for interaction scenarios requiring real-time feedback. Additionally, when customizing systems for specific industries, model training can only be achieved by collecting and labeling a large amount of industry-specific speech data. Labeling costs and technical barriers are high, the process is lengthy, and it lacks versatility and flexibility. Summary of the Invention

[0005] Therefore, the purpose of this invention is to propose a hybrid speech recognition method and system based on an AI large-scale model architecture. This method retains the advantages of traditional ASR systems, such as flexible customization, streaming output, and low cost, while integrating the powerful modeling capabilities of AI large-scale models to improve overall recognition accuracy. It combines the advantages of traditional ASR systems in terms of flexible customization for vertical domains and streaming recognition with the advantages of LLM-driven ASR systems in terms of recognition accuracy and model capability improvement. It utilizes massive amounts of unlabeled speech data for efficient pre-training and achieves a more effective combination of acoustic and linguistic information, effectively reducing the data labeling cost and technical threshold for system customization. Through self-supervised learning, modular architecture, joint training, and multi-output head design, it achieves a unification of streaming and high-precision recognition methods. While ensuring the high-precision recognition capability of AI large-scale models, it supports streaming, low-latency online recognition requirements, improving the real-time user interaction experience. Furthermore, through multimodal fusion of speech and text, it supports the addition of user commands and contextual information, effectively improving domain adaptability and user experience, making it suitable for various scenarios such as human-computer interaction, voice assistants, and real-time transcription.

[0006] This invention provides a hybrid speech recognition method based on an AI large model architecture, comprising the following steps:

[0007] S1. Construct a hybrid ASR system (based on decoder-only or encoder-decoder multimodal speech / text self-supervised model) that includes a traditional ASR system (acoustic model + language model + decoder) and an AI large model; set a dual-ended output head in the core model of the hybrid system. One end of the dual-ended output head is used to directly output acoustic recognition units (such as CTC / pinyin / word / phoneme), which are independently embedded as acoustic models in the traditional ASR system; the other end outputs text token sequences, supporting LLM / ASR end-to-end capabilities;

[0008] S2. During the model training phase, the AI ​​model is pre-trained using a self-supervised learning method based on a large amount of unlabeled audio data to mine the features of audio signals. The AI ​​model is then fine-tuned using supervised SFT with limited high-quality labeled speech data. Furthermore, human feedback reinforcement learning (RLHF) is introduced to further optimize the recognition accuracy.

[0009] Specifically, human feedback reinforcement learning (RLHF) can achieve precise alignment between large language models and human preferences.

[0010] S3. Train the text output using cross-entropy loss in the base layer of the AI ​​model; introduce an acoustic modeling branch in the intermediate layer or auxiliary head, and train the acoustic output using a CTC loss function suitable for block alignment; combine the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion, while improving the performance of end-to-end and traditional decoding models.

[0011] Cross-entropy loss is commonly used to measure the difference between two probability distributions and is widely applied in machine learning, especially in classification problems. In the joint loss and hierarchical output of this invention, cross-entropy loss is used to train the text output in the base layer of the large AI model.

[0012] Specifically, acoustic modeling is a key component of speech recognition systems. The purpose of acoustic modeling is to establish a mapping relationship between the acoustic features of speech signals and language information, converting the speech waveform spoken by humans into information that computers can understand and process.

[0013] S4. During the model training or inference phase, audio segmentation and KV-cache technology are used to achieve efficient streaming recognition. When audio segments are input into the decoding model, the previous activation state is retained, supporting recognition results while speaking, which greatly reduces latency and improves user experience. After speaking, the AI ​​big model quickly completes subsequent autoregressive text inference based on all cached information and outputs high-precision recognized text.

[0014] Specifically, the audio segmentation method divides an audio segment into multiple segments for acoustic model training, with the aim of enabling the acoustic model to support streaming speech recognition (recognition while the user speaks, reducing latency).

[0015] In traditional computing, all key-value information needs to be recalculated each time a new token is generated. In the fields of speech recognition and NLP, with the increase in model size and the higher requirements for response speed, key-value caching technology has emerged. Key-value caching (KV-cache) is based on the Transformer self-attention mechanism. It stores the calculated key-value pairs during inference, and when a new token arrives, only the current key-value pairs are calculated. After concatenating with historical data, attention calculation is performed. By optimizing the attention mechanism calculation of the Transformer architecture, KV caching significantly improves the model inference efficiency.

[0016] The working mechanism of key-value caching is as follows:

[0017] Initialization: Prepare empty key-value caches according to batch quantity, maximum content length, and information dimension;

[0018] First unit: After the first token is input, calculate the key-value pair and store it in the corresponding cache location;

[0019] Subsequent unit: Only calculate the new token key-value pair, concatenate it with historical data in the cache to complete the calculation, and update the cache;

[0020] Cache management: When the preset length is reached, updates are performed using methods such as sliding windows, and the oldest data is deleted to accommodate new content.

[0021] The advantages of key-value caching are as follows:

[0022] Reduce computational load: Reusing historical results significantly reduces computational load when processing long sequences;

[0023] Improve speed: Reduce redundant calculations and lower latency, suitable for real-time response scenarios;

[0024] Optimize memory: Save memory overhead for repetitive calculations and achieve overall optimization.

[0025] Preferably, in terms of domain customization and hybrid output, the present invention supports the fusion of model output with language model scores such as traditional N-grams, and combines external domain text to quickly customize domain language models. By combining multi-source scores through decoders, there is no need to re-annotate a large amount of speech data, resulting in a short cycle and low cost, which meets the actual needs of vertical industries.

[0026] Preferably, in terms of personalized input and context enhancement, the input layer of the present invention can introduce user-personalized instructions and contextual information to provide the capability for targeted recognition and contextual adaptation, thereby improving the recognition accuracy and intelligence level in complex scenarios.

[0027] Furthermore, the supervised fine-tuning method for the large AI model in step S2 includes the following steps:

[0028] S201. Perform unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities; for example, GPT-3, by processing hundreds of millions of text data, grasps the basic rules of natural language, providing a solid foundation for subsequent optimization.

[0029] S202. Construct a dedicated labeled dataset around the target task. The data volume of the dedicated labeled dataset is smaller than that of the pre-training dataset, but it is highly consistent with the application scenario. For example, in the medical field, it is necessary to collect texts such as medical records and diagnostic reports, and label structured information such as disease classification and symptom extraction.

[0030] S203. Use labeled data to train the pre-trained model in a targeted manner, and continuously optimize the loss function to adjust the model parameters in line with the task requirements, thereby gradually improving the prediction accuracy in specific scenarios.

[0031] S204. Use an independent test set that was not used in training to test the model performance, and evaluate it quantitatively using accuracy and recall metrics. If the quantitative evaluation effect does not meet expectations, improve it by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics meets expectations.

[0032] Supervised Fine-Tuning (SFT), a core technology for improving AI model performance, achieves precise upgrades to model capabilities by injecting domain knowledge into pre-trained models. SFT starts with pre-trained models that have already built their basic capabilities through massive amounts of unlabeled data; for example, language models have mastered grammatical structure and semantic logic. For specific tasks, high-quality labeled datasets need to be constructed, with each sample containing input content and a corresponding standard answer. Subsequently, by minimizing the error between the model's predictions and the labeled results, optimization algorithms such as gradient descent are used to iteratively update the model parameters, enabling the model to learn task-specific rules and ultimately achieve functional customization.

[0033] Furthermore, the method for further optimizing recognition accuracy by introducing human feedback reinforcement learning (RLHF) in step S2 includes the following steps:

[0034] S211. Collect human feedback data through a multi-dimensional annotation system, including: giving star ratings to the content generated by the AI ​​large model and ranking multiple candidate answers according to their quality; the human feedback data collected by the annotation system constitutes the basic corpus for subsequent model training.

[0035] S212. Construct a reward model, use the collected feedback data as supervised learning samples to train the reward model, the reward model can quantitatively evaluate the quality of different outputs and learn the inherent logic of human judgment of text quality.

[0036] S213. Set an initial strategy for the AI ​​large model, enabling it to interact with the virtual environment and generate output content; the reward model scores the output results and assigns corresponding reward values; using the Proximal Policy Optimization (PPO) reinforcement learning algorithm, adjust its own strategy based on the reward signal, optimize the content generation logic, and achieve iterative update of strategy optimization:

[0037] S214. After the strategy is updated, human feedback data of the new output content is continuously collected, the reward model is retrained using the new data, and the AI ​​large model strategy is iterated again based on the optimized reward signal. Through a closed-loop cyclic improvement mechanism, the model output is continuously improved to better match human values ​​and usage preferences.

[0038] Furthermore, the features of the mined audio signal in step S2 include: masking prediction, frame restoration, and prediction of the next speech segmentation unit;

[0039] The masking prediction method includes: when processing audio data, randomly masking some audio segments, and the model predicting the content of the masked parts based on the unmasked audio information; in practice, the audio spectrogram can be processed, and some frequencies or time segments can be randomly selected for masking, and then the model can learn to predict the audio features of these masked areas, such as the frequency and amplitude information of the audio; through a large number of such trainings, the large model can better understand the structure and features of audio and improve its ability to process various audio situations;

[0040] The frame restoration method includes: dividing the audio into multiple frames at certain time intervals, performing special processing on some frames, such as adding noise or changing the amplitude, so that the model learns to restore these processed frames to the original normal frames; when dividing the audio frames, setting appropriate frame length and frame shift according to the characteristics of the audio, and then performing interference operations on the selected frames. The model learns continuously how to remove these interferences and restore the features of the original frames; this can enhance the ability to capture and restore audio details and improve recognition performance in complex audio environments.

[0041] The method for predicting the next speech segmentation unit includes: when processing an audio segment, the model predicts the next speech segmentation unit based on the information of previously processed speech segmentation units; the model performs segmentation processing on the input audio sequentially, and after processing the current segmentation unit, it combines existing contextual information (including previous segmentation results, overall audio features, etc.) to predict the next most likely speech segmentation unit. Through extensive training of this kind, the large model can learn the semantic and grammatical coherence of speech, improving the understanding and recognition accuracy of speech content.

[0042] Furthermore, the method for training the text output using cross-entropy loss in step S3 includes:

[0043] For speech recognition, which requires identifying words within a speech segment, the expression for the cross-entropy loss is:

[0044]

[0045] in, The probability distribution representing the real text labels, then It refers to the probability of each possible word (i.e., the category of the recognition result) appearing in a real situation; for a specific speech segment, only one word is actually spoken, so The value is 1 at the correct word position and 0 at the rest.

[0046] This represents the probability distribution of the text predicted by the model. After a series of calculations, the model outputs a probability value for each possible recognition result, and the distribution formed by these probability values ​​is the probability distribution. For example, when a model predicts the text corresponding to a piece of speech, it assigns a probability to each possible word, and these probabilities constitute... distributed;

[0047] n represents the number of all possible categories, which in the context of speech recognition is the total number of words in the vocabulary, i.e., the number of all possible recognition results that the model can output; if there are 1000 words in the vocabulary, then n = 1000.

[0048] By calculating the predicted text probability distribution Probability distribution of real text labels The model continuously adjusts its parameters to minimize the cross-entropy loss, making the predicted distribution more closely resemble the real distribution, thereby improving the accuracy of the speech recognition system in text recognition.

[0049] Furthermore, the method for training the acoustic output using a CTC loss function suitable for block alignment in step S3 includes:

[0050] Constructing the CTC loss function:

[0051]

[0052] in, Calculated using a forward-backward algorithm, i.e., given an input sequence The output sequence is obtained. The probability is as follows:

[0053]

[0054] in, This refers to the total number of training samples, such as 1000 training speech segments in speech recognition, covering different sentences, speakers, and environmental noise;

[0055] For the first Each input sample is a sequence of speech after preprocessing and feature extraction, such as MFCC, FBANK and other feature sequences;

[0056] for The corresponding real label sequence, that is, the text content actually expressed by the speech;

[0057] The intermediate sequence predicted by the model is longer and contains whitespace symbols ("-"), which solves the problem of speech-text alignment.

[0058] For a given At that time, the model predicts The probability reflects the likelihood of prediction;

[0059] For a given At that time, the real label is obtained. The probability is obtained by summing the probabilities of the relevant intermediate sequences. The model minimizes the loss function. Improve recognition accuracy.

[0060] Specifically, Connectionist Temporal Classification (CTC) is an algorithm for handling sequence-to-sequence problems. In the scheme of this invention, which introduces a dedicated acoustic modeling branch in the intermediate layer or auxiliary head, the CTC loss function enables the new branch to handle the complex relationship between speech and text, thereby improving the performance of the speech recognition system.

[0061] Furthermore, the method for introducing the acoustic modeling branch in step S3 includes:

[0062] The acoustic characteristics of speech are analyzed and modeled. The speech signal is divided into short frames in time sequence, and the feature parameters of each frame are extracted. The feature parameters include Mel-frequency cepstral coefficients (MFCC) and linear predictive cepstral coefficients (LPCC). The feature parameters can reflect the acoustic characteristics of speech.

[0063] The acoustic model, based on the feature parameters, learns the acoustic performance patterns of different speech units (such as phonemes, syllables, etc.) under various pronunciation conditions through statistical methods or machine learning algorithms.

[0064] This invention also provides a hybrid speech recognition system based on an AI large model architecture, which performs the hybrid speech recognition method based on an AI large model architecture as described above, including:

[0065] Model architecture fusion module: used to build a hybrid system of ASR system including traditional ASR system and AI large model; a dual-end output head is set in the core model of the hybrid system. One end of the dual-end output head is used to directly output acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs text token sequence, supporting LLM / ASR end-to-end capability;

[0066] Multimodal self-supervised training module: During the model training phase, it is used to pre-train the AI ​​model based on a large amount of unlabeled audio data through self-supervised learning, mine the features of audio signals, and perform supervised fine-tuning of the model using limited high-quality labeled speech data; and introduces human feedback reinforcement learning (RLHF) to further optimize the recognition accuracy.

[0067] Joint Loss and Hierarchical Output Module: Used to train text output using cross-entropy loss in the base layer of the AI ​​model; introduces acoustic modeling branches in intermediate layers or auxiliary heads, and trains acoustic output using a CTC loss function suitable for block alignment; combines the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion.

[0068] Streaming Inference and Key-Value Caching Module: This module utilizes audio segmentation and key-value caching technology during model training or inference to achieve efficient streaming recognition. When audio segments are input into the decoding model, the preceding activation states are retained. The solution supports simultaneous speaking and output of recognition results, significantly reducing latency and improving user experience. After speaking, the model can quickly complete subsequent autoregressive text inference based on all cached information, outputting high-precision recognized text.

[0069] This invention employs a dual-output / hybrid architecture to achieve compatibility between high precision and a streaming experience. It leverages the advantages of large AI models, such as high end-to-end recognition rate and strong semantic understanding, while retaining the capabilities of traditional ASR systems in streaming inference, real-time feedback, and flexible domain customization. Pre-training with massive amounts of unlabeled speech data significantly reduces reliance on labeled data, lowering the industry customization threshold and cost, and facilitating faster adaptation and deployment in various industry scenarios. The innovative multi-output head architecture and joint loss training enhance model information coupling and generalization capabilities, effectively overcoming model mismatch and cumulative error issues common in traditional solutions. The use of KV caching and block inference allows for seamless switching between speaking and display, low-latency streaming speech recognition, and high-precision post-processing, greatly improving the user experience. Support for dual-track output and multimodal access facilitates subsequent integration and expansion with other downstream AI tasks (such as semantic understanding and intelligent question answering), extending the application scope. Furthermore, the domain optimization cycle of this invention is short, requiring minimal manual annotation, thus improving project delivery efficiency.

[0070] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the hybrid speech recognition method based on the AI ​​large model architecture described above.

[0071] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the hybrid speech recognition method based on the AI ​​large model architecture described above.

[0072] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0073] This invention provides a hybrid speech recognition method and system based on an AI large-scale model architecture. It retains the advantages of traditional ASR systems, such as flexible customization, streaming output, and low cost, while integrating the powerful modeling capabilities of AI large-scale models to improve overall recognition accuracy. It combines the advantages of traditional ASR systems in terms of flexible customization for vertical domains and streaming recognition with the advantages of LLM-driven ASR systems in terms of recognition accuracy and model capability improvement. It utilizes unlabeled massive speech data for efficient pre-training and achieves a more effective combination of acoustic and linguistic information, effectively reducing the data labeling cost and technical threshold for system customization. Through self-supervised learning, modular architecture, joint training, and multi-output head design, it achieves a unification of streaming and high-precision recognition methods. While ensuring the high-precision recognition capability of AI large-scale models, it supports streaming, low-latency online recognition requirements, improving the real-time user interaction experience. Furthermore, through multimodal fusion of speech and text, it supports the addition of user commands and contextual information, effectively improving domain adaptability and user experience. It is suitable for various scenarios such as human-computer interaction, voice assistants, and real-time transcription, and has broad application prospects. Attached Figure Description

[0074] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0075] In the attached diagram:

[0076] Figure 1 This is a diagram of a traditional hybrid speech recognition system framework;

[0077] Figure 2 The existing LLM-based ASR system architecture diagram;

[0078] Figure 3 Example diagram of existing LLM-based ASR autoregressive decoding;

[0079] Figure 4 This is a diagram of a self-supervised learning network based on an AI large model architecture, as described in an embodiment of the present invention.

[0080] Figure 5 This is an architecture diagram of a hybrid speech recognition system based on an AI large model architecture, according to an embodiment of the present invention.

[0081] Figure 6 This is a flowchart of a hybrid speech recognition method based on an AI large model architecture according to the present invention;

[0082] Figure 7 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation

[0083] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and products consistent with some aspects of this disclosure as detailed in the appended claims.

[0084] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0085] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0086] The embodiments of the present invention will be described in further detail below.

[0087] This invention provides a hybrid speech recognition method based on an AI large model architecture. (See also...) Figure 6 As shown, it includes the following steps:

[0088] S1. Construct a hybrid ASR system that includes a traditional ASR system and an AI large model; set a dual-end output head in the core model of the hybrid system. One end of the dual-end output head is used to directly output the acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs a text token sequence, supporting LLM / ASR end-to-end capabilities.

[0089] S2. During the model training phase, the AI ​​model is pre-trained using a self-supervised learning method based on a large amount of unlabeled audio data to mine the features of audio signals. The AI ​​model is then fine-tuned using supervised SFT with limited high-quality labeled speech data. Furthermore, human feedback reinforcement learning (RLHF) is introduced to further optimize the recognition accuracy.

[0090] The neural network architecture corresponding to the training phase of this model is as follows: Figure 4 As shown;

[0091] The method for supervised fine-tuning of large AI models includes the following steps:

[0092] S201. Perform unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities; for example, GPT-3, by processing hundreds of millions of text data, grasps the basic rules of natural language, providing a solid foundation for subsequent optimization.

[0093] S202. Construct a dedicated labeled dataset around the target task. The data volume of the dedicated labeled dataset is smaller than that of the pre-training dataset, but it is highly consistent with the application scenario. For example, in the medical field, it is necessary to collect texts such as medical records and diagnostic reports, and label structured information such as disease classification and symptom extraction.

[0094] S203. Use labeled data to train the pre-trained model in a targeted manner, and continuously optimize the loss function to adjust the model parameters in line with the task requirements, thereby gradually improving the prediction accuracy in specific scenarios.

[0095] S204. Use an independent test set that was not used in training to test the model performance, and evaluate it quantitatively using accuracy and recall metrics. If the quantitative evaluation effect does not meet expectations, improve it by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics meets expectations.

[0096] The method of introducing human feedback reinforcement learning (RLHF) to further optimize recognition accuracy includes the following steps:

[0097] S211. Collect human feedback data through a multi-dimensional annotation system, including: giving star ratings to the content generated by the AI ​​large model and ranking multiple candidate answers according to their quality; the human feedback data collected by the annotation system constitutes the basic corpus for subsequent model training.

[0098] S212. Construct a reward model, use the collected feedback data as supervised learning samples to train the reward model, the reward model can quantitatively evaluate the quality of different outputs and learn the inherent logic of human judgment of text quality.

[0099] S213. Set an initial strategy for the AI ​​large model, enabling it to interact with the virtual environment and generate output content; the reward model scores the output results and assigns corresponding reward values; using the Proximal Policy Optimization (PPO) reinforcement learning algorithm, adjust its own strategy based on the reward signal, optimize the content generation logic, and achieve iterative update of strategy optimization:

[0100] S214. After the strategy is updated, human feedback data of the new output content is continuously collected, the reward model is retrained using the new data, and the AI ​​large model strategy is iterated again based on the optimized reward signal. Through a closed-loop cyclic improvement mechanism, the model output is continuously improved to better match human values ​​and usage preferences.

[0101] The features of audio signals include: masking prediction, frame reconstruction, and prediction of the next speech segmentation unit;

[0102] The masking prediction method includes: when processing audio data, randomly masking some audio segments, and the model predicting the content of the masked parts based on the unmasked audio information; in practice, the audio spectrogram can be processed, and some frequencies or time segments can be randomly selected for masking, and then the model can learn to predict the audio features of these masked areas, such as the frequency and amplitude information of the audio; through a large number of such trainings, the model can better understand the structure and features of audio and improve its ability to process various audio situations.

[0103] The frame restoration method includes: dividing the audio into multiple frames at certain time intervals, performing special processing on some frames, including adding noise and changing the amplitude, so that the model can learn to restore these processed frames to the original normal frames; when dividing the audio frames, setting appropriate frame length and frame shift according to the characteristics of the audio, and then performing interference operations on the selected frames. The model learns continuously how to remove these interferences and restore the features of the original frames; enhancing the ability to capture and restore audio details and improving recognition performance in complex audio environments.

[0104] Methods for predicting the next speech segmentation unit include: when processing an audio segment, the model predicts the next speech segmentation unit based on the information of previously processed speech segmentation units; the model performs segmentation processing on the input audio sequentially, and after processing the current segmentation unit, it combines existing contextual information, including previous segmentation results and the overall features of the audio, to predict the next most likely speech segmentation unit. Through extensive training of this kind, the model can learn the semantic and grammatical coherence of speech, improving the understanding and recognition accuracy of speech content.

[0105] S3. Train the text output using cross-entropy loss in the base layer of the AI ​​model; introduce an acoustic modeling branch in the intermediate layer or auxiliary head, and train the acoustic output using a CTC loss function suitable for block alignment; combine the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion, while improving the performance of end-to-end and traditional decoding models.

[0106] Methods for training text output using cross-entropy loss include:

[0107] For speech recognition, which requires identifying words within a speech segment, the expression for the cross-entropy loss is:

[0108]

[0109] in, The probability distribution representing the real text labels, then It refers to the probability of each possible word (i.e., the category of the recognition result) appearing in a real situation; for a specific speech segment, only one word is actually spoken, so The value is 1 at the correct word position and 0 at the rest (this is a common way of representing the true label using one-hot encoding). For example, if the correct recognition result is "apple", then in the probability distribution vector composed of all possible words, the position corresponding to "apple" has a probability of 1, and the positions of other words have a probability of 0.

[0110] This represents the probability distribution of the text predicted by the model. In this hybrid speech recognition system, after a series of calculations, the model outputs a probability value for each possible recognition result. The distribution formed by these probability values ​​is... For example, when a model predicts the text corresponding to a piece of speech, it assigns a probability to each possible word, and these probabilities constitute... distributed;

[0111] n represents the number of all possible categories, which in the context of speech recognition is the total number of words in the vocabulary, i.e., the number of all possible recognition results that the model can output; if there are 1000 words in the vocabulary, then n = 1000.

[0112] By calculating the predicted text probability distribution Probability distribution of real text labels By using cross-entropy loss, the model continuously adjusts its parameters to make the prediction results as close as possible to the real situation, making the predicted distribution more consistent with the real distribution, thus improving the accuracy of the speech recognition system in text recognition.

[0113] Methods for training acoustic output using a CTC loss function suitable for block alignment include:

[0114] Constructing the CTC loss function:

[0115]

[0116] in, Calculated using a forward-backward algorithm, i.e., given an input sequence The output sequence is obtained. The probability is as follows:

[0117]

[0118] in, This refers to the total number of training samples, such as 1000 training speech segments in speech recognition, covering different sentences, speakers, and environmental noise;

[0119] For the first Each input sample is a sequence of speech after preprocessing and feature extraction, such as MFCC, FBANK and other feature sequences;

[0120] for The corresponding real label sequence, that is, the text content actually expressed by the speech;

[0121] The intermediate sequence predicted by the model is longer and contains whitespace symbols ("-"), which solves the problem of speech-text alignment.

[0122] For a given At that time, the model predicts The probability reflects the likelihood of prediction;

[0123] For a given At that time, the real label is obtained. The probability is obtained by summing the probabilities of the relevant intermediate sequences. The model minimizes the loss function. Improve recognition accuracy.

[0124] This embodiment introduces a dedicated acoustic modeling branch in the intermediate layer or auxiliary head. The CTC loss function allows the new branch to handle the complex relationship between speech and text, thereby improving the performance of the speech recognition system.

[0125] The methods that introduce acoustic modeling as a branch include:

[0126] The acoustic characteristics of speech are analyzed and modeled. The speech signal is divided into short frames in time sequence, and the feature parameters of each frame are extracted. The feature parameters include Mel-frequency cepstral coefficients (MFCC) and linear predictive cepstral coefficients (LPCC). The feature parameters can reflect the acoustic characteristics of speech.

[0127] The acoustic model, based on the aforementioned feature parameters, learns the acoustic performance patterns of different speech units (including phonemes, syllables, etc.) under various pronunciation conditions through statistical methods or machine learning algorithms.

[0128] When recognizing the word "apple," acoustic modeling analyzes the acoustic feature changes of each phoneme that makes up the pronunciation of "apple" in actual speech. For example, when different people say "apple," due to differences in pronunciation habits, speaking speed, and intonation, the speech waveform and acoustic features will also differ. The acoustic model must capture these patterns of change and establish a reliable mapping relationship so that the system can accurately convert speech into corresponding text. In your mention of "introducing a dedicated acoustic modeling branch in the intermediate or auxiliary head," this means adding a part to the model architecture that focuses on learning and processing speech acoustic features, further improving the speech recognition system's ability to process speech signals and its recognition accuracy.

[0129] S4. During the model training or inference phase, audio segmentation and KV caching technology are used to achieve efficient streaming recognition. When audio segments are input into the decoding model, the previous activation state is retained, supporting recognition results while speaking. After speaking, the AI ​​big model quickly completes subsequent autoregressive text inference based on all cached information and outputs high-precision recognized text.

[0130] Audio segmentation divides an audio segment into multiple segments for acoustic model training, with the aim of enabling the acoustic model to support streaming speech recognition (recognizing speech as the user speaks, reducing latency).

[0131] In traditional computation, all key-value information needs to be recalculated each time a new token is generated. Key-value caching, on the other hand, stores already calculated key-value pairs during inference. When a new token arrives, only the current key-value pair is calculated, concatenated with historical data, and then attention is calculated. Key-value caching significantly improves model inference efficiency by optimizing the attention mechanism computation of the Transformer architecture.

[0132] This invention also provides a hybrid speech recognition system based on an AI large model architecture, which executes the hybrid speech recognition method based on an AI large model architecture as described above, including:

[0133] Model architecture fusion module: used to build a hybrid system of ASR system including traditional ASR system and AI large model; a dual-end output head is set in the core model of the hybrid system. One end of the dual-end output head is used to directly output acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs text token sequence, supporting LLM / ASR end-to-end capability;

[0134] Multimodal self-supervised training module: During the model training phase, it is used to pre-train the AI ​​model based on a large amount of unlabeled audio data through self-supervised learning, mine the features of audio signals, and perform supervised fine-tuning of the model using limited high-quality labeled speech data; and introduces human feedback reinforcement learning (RLHF) to further optimize the recognition accuracy.

[0135] Joint Loss and Hierarchical Output Module: Used to train text output using cross-entropy loss in the base layer of the AI ​​model; introduces acoustic modeling branches in intermediate layers or auxiliary heads, and trains acoustic output using a CTC loss function suitable for block alignment; combines the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion.

[0136] Streaming Inference and Key-Value Caching Module: This module utilizes audio segmentation and key-value caching technology during model training or inference to achieve efficient streaming recognition. When audio segments are input into the decoding model, the preceding activation states are retained. The solution supports simultaneous speaking and output of recognition results, significantly reducing latency and improving user experience. After speaking, the model can quickly complete subsequent autoregressive text inference based on all cached information, outputting high-precision recognized text.

[0137] Figure 5 The architecture of the hybrid speech recognition system based on the AI ​​large model architecture of this embodiment is shown.

[0138] This invention also provides a computer device. Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 7 As shown, the computer device includes: an input system 23, an output system 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the hybrid speech recognition method based on the AI ​​large model architecture provided in the above embodiments; wherein the input system 23, the output system 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0139] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the hybrid speech recognition method based on the AI ​​large model architecture described in this embodiment of the invention. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The input system 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; the output system 24 may include display devices such as a display screen.

[0141] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned hybrid speech recognition method based on the AI ​​large model architecture.

[0142] The computer equipment provided above can be used to execute the hybrid speech recognition method based on the AI ​​large model architecture provided in the above embodiments, and has corresponding functions and beneficial effects.

[0143] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the hybrid speech recognition method based on the AI ​​large model architecture provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape systems; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0144] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the hybrid speech recognition method based on the AI ​​large model architecture described in the above embodiments, but can also perform related operations in the hybrid speech recognition method based on the AI ​​large model architecture provided in any embodiment of the present invention.

[0145] The technical solution of the present invention has been described in conjunction with preferred embodiments. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from these changes or substitutions will all fall within the scope of protection of the present invention.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid speech recognition method based on an AI large-scale model architecture, characterized in that, Includes the following steps: S1. Construct a hybrid ASR system that includes a traditional ASR system and an AI large model; set a dual-end output head in the core model of the hybrid system. One end of the dual-end output head is used to directly output the acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs a text token sequence, supporting LLM / ASR end-to-end capabilities. S2. During the model training phase, the AI ​​model is pre-trained using a self-supervised learning method based on a large amount of unlabeled audio data to mine the features of the audio data. The AI ​​model is then fine-tuned using supervised SFT with limited high-quality labeled speech data. Furthermore, human feedback reinforcement learning (RLHF) is introduced to further optimize the recognition accuracy. S3. Train the text output using cross-entropy loss in the base layer of the AI ​​model; introduce branches of the acoustic model in the intermediate layer or auxiliary head, and train the acoustic output using a CTC loss function suitable for block alignment. By combining the CTC loss function with the cross-entropy loss during training, acoustic and textual information fusion is achieved, while simultaneously improving the performance of end-to-end and traditional decoding models. S4. During the model training or inference phase, audio segmentation and KV caching technology are used to achieve efficient streaming recognition. When audio segments are input into the decoding model, the previous activation state is retained, supporting recognition results while speaking. After speaking, the AI ​​big model quickly completes subsequent autoregressive text inference based on all cached information and outputs high-precision recognized text.

2. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The supervised fine-tuning method for the large AI model in step S2 includes: S201. Perform unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities for large AI models. S202. Construct a dedicated labeled dataset around the target task. The data volume of the dedicated labeled dataset is smaller than the pre-training scale, but it is highly consistent with the application scenario. S203. Use labeled data to train the pre-trained model in a targeted manner, and continuously optimize the loss function to adjust the model parameters in line with the task requirements, thereby gradually improving the prediction accuracy in the application scenario. S204. Use an independent test set that was not used in training to test the model performance, and evaluate it quantitatively using accuracy and recall metrics. If the quantitative evaluation effect does not meet expectations, improve it by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics meets expectations.

3. The hybrid speech recognition method based on an AI large model architecture according to claim 2, characterized in that, The method for further optimizing recognition accuracy by introducing human feedback reinforcement learning (RLHF) in step S2 includes: S211. Collect human feedback data through a multi-dimensional annotation system, including: giving star ratings to the content generated by the AI ​​large model and ranking multiple candidate answers according to their quality; the human feedback data collected by the annotation system constitutes the basic corpus for subsequent model training. S212. Construct a reward model, use the collected feedback data as supervised learning samples to train the reward model, the reward model can quantitatively evaluate the quality of different outputs and learn the inherent logic of human judgment of text quality. S213. Set an initial strategy for the AI ​​large model, enabling it to interact with the virtual environment and generate output content; the reward model scores the output results and assigns corresponding reward values; the reinforcement learning algorithm is optimized using a proximal policy, adjusting its own strategy based on reward signals, optimizing the content generation logic, and achieving iterative updates to the strategy optimization: S214. After the strategy is updated, human feedback data of the new output content is continuously collected, the reward model is retrained using the new data, and the AI ​​large model strategy is iterated again based on the optimized reward signal. Through a closed-loop cyclic improvement mechanism, the model output is continuously improved to better match human values ​​and usage preferences.

4. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The features of the audio signal mined in step S2 include: masking prediction, frame restoration, and prediction of the next speech segmentation unit; The masking prediction method includes: when processing audio data, randomly masking some audio segments, and the model predicting the content of the masked parts based on the unmasked audio information; in actual operation, the audio spectrogram is processed, some frequencies or time segments are randomly selected for masking, and then the model is allowed to learn to predict the audio features of these masked areas, including the frequency and amplitude information of the audio. The frame restoration method includes: dividing the audio into multiple frames according to a certain time interval, performing special processing on some frames, and allowing the model to learn to restore these processed frames to the original normal frames; when dividing the audio frames, setting an appropriate frame length and frame shift according to the characteristics of the audio, and then performing interference operations on the selected frames. The model learns continuously how to remove these interferences and restore the features of the original frames. The method for predicting the next speech segmentation unit includes: when the model processes an audio segment, it predicts the next speech segmentation unit based on the information of the speech segmentation units that have already been processed; the model performs segmentation processing on the input audio segment sequentially, and after processing the current segmentation unit, it predicts the next most likely speech segmentation unit by combining the existing context information.

5. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for training text output using cross-entropy loss in step S3 includes: For speech recognition, which requires identifying words within a speech segment, the expression for the cross-entropy loss is: in, The probability distribution representing the real text labels. It refers to the probability of each possible word appearing in a real situation. The value is 1 at the correct word position and 0 at other positions; This represents the probability distribution of the text predicted by the model. After a series of calculations, the model outputs a probability value for each possible recognition result, and the distribution formed by these probability values ​​is the probability distribution. ; n represents the number of all possible categories, which in the context of speech recognition is the total number of words in the vocabulary, i.e., the number of all possible recognition results that the model can output; By calculating the predicted text probability distribution Probability distribution of real text labels The model continuously adjusts its parameters to minimize the cross-entropy loss between predictions, making the predictions as close to the actual situation as possible.

6. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for training the acoustic output using a CTC loss function suitable for block alignment in step S3 includes: Constructing the CTC loss function: in, Calculated using a forward-backward algorithm, i.e., given an input sequence The output sequence is obtained. The probability is as follows: in, The total number of training samples includes 1000 training speech segments from speech recognition, covering different sentences, speakers, and environmental noise; For the first Each input sample is a sequence of speech after preprocessing and feature extraction, including MFCC and FBANK feature sequences; for The corresponding real label sequence, that is, the text content actually expressed by the speech; The intermediate sequence predicted by the model is longer and contains whitespace symbols ("-"), which solves the problem of speech-text alignment. For a given At that time, the model predicts The probability reflects the likelihood of prediction; For a given At that time, the real label is obtained. The probability is obtained by summing the probabilities of the relevant intermediate sequences. The model minimizes the loss function. Improve recognition accuracy.

7. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for introducing the acoustic modeling branch in step S3 includes: The acoustic characteristics of speech are analyzed and modeled by dividing the speech signal into short frames in chronological order and extracting the feature parameters of each frame. The feature parameters include Mel frequency cepstral coefficients and linear prediction cepstral coefficients. These feature parameters can reflect the acoustic characteristics of speech. The acoustic model, based on the feature parameters, learns the acoustic performance patterns of different speech units under various pronunciation conditions through statistical methods or machine learning algorithms.

8. A hybrid speech recognition system based on an AI large-scale model architecture, characterized in that, Performing the hybrid speech recognition method based on an AI large model architecture as described in any one of claims 1-7, comprising: Model architecture fusion module: used to build a hybrid system of ASR system including traditional ASR system and AI large model; a dual-end output head is set in the core model of the hybrid system. One end of the dual-end output head is used to directly output acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs text token sequence, supporting LLM / ASR end-to-end capability; Multimodal self-supervised training module: During the model training phase, it is used to pre-train the AI ​​model based on a large amount of unlabeled audio data through self-supervised learning, mine the features of audio signals, and perform supervised fine-tuning of the model using limited high-quality labeled speech data; and introduces human feedback reinforcement learning (RLHF) to further optimize the recognition accuracy. Joint Loss and Hierarchical Output Module: Used to train text output using cross-entropy loss in the base layer of the AI ​​model; introduces acoustic modeling branches in intermediate layers or auxiliary heads, and trains acoustic output using a CTC loss function suitable for block alignment; combines the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion. Streaming Inference and KV Caching Module: Used during model training or inference phases, this module employs audio segmentation and KV caching technology to achieve efficient streaming recognition. When audio segments are input into the decoding model, the preceding activation states are retained. The solution supports simultaneous speaking and output of recognition results, greatly reducing latency and improving user experience. After speaking, the model quickly completes subsequent autoregressive text inference based on all cached information, outputting high-precision recognized text.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the hybrid speech recognition method based on the AI ​​large model architecture as described in any one of claims 1-7.

10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the hybrid speech recognition method based on an AI large model architecture as described in any one of claims 1-7.

Images