Hybrid speech recognition method and system based on AI large model architecture

Abstract

The invention provides a mixed speech recognition method and system based on an AI large model architecture. The method comprises the following steps: constructing a mixed system comprising a traditional ASR system and an ASR system of an AI large model; based on a large amount of unlabeled audio data, pre-training the large model through self-supervised learning, mining audio signal features, and performing supervised fine tuning on the AI large model; using cross entropy loss to train text output in the base layer; an acoustic modeling branch is introduced into the middle layer or the auxiliary head, and a CTC loss function is adopted to train acoustic output; combining a CTC loss function with cross entropy loss training to realize acoustic and text information fusion; in a model training or reasoning stage, an audio partitioning mode and a KV cache technology are adopted to realize efficient streaming identification; and the recognition result can be obtained while speaking, and after the calling is finished, the large model quickly completes the subsequent autoregression text reasoning based on all cache information, and outputs a high-precision recognition text. According to the invention, compatibility of high precision and streaming experience is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, speech recognition and natural language processing, in particular, to a hybrid speech recognition technology combining an AI large model and a traditional automatic speech recognition (ASR) system architecture; specifically, to a hybrid speech recognition method and system based on an AI large model architecture. BACKGROUND

[0002] With the rapid popularization of voice interaction in the fields of mobile Internet, intelligent terminals, office scenarios, etc., ASR (speech recognition) technology has become the basis of audio understanding. The traditional ASR system adopts the modeling method of separating acoustic model and language model, and the system architecture is as shown in Figure 1 The acoustic model needs to be supervised trained by a large amount of labeled data, and the language model generally adopts N-gram technology and only needs text data. The acoustic model and the language model are fused by a decoder to output the final recognition result. This architecture facilitates the use of domain text to enhance the language model, achieving the purpose of quickly adapting to specific scenarios and reducing customization costs. However, the acoustic model and the language model of the traditional ASR system are independently trained, which will lead to insufficient interaction of information, easy accumulation of errors, and difficulty in fully utilizing massive unlabeled audio resources for the acoustic model, greatly limiting the improvement of recognition effect.

[0003] In recent years, with the development of large language model (LLM) technology, end-to-end ASR solutions have gradually emerged. For example, a text AI large model is expanded into a speech model through an adaptive input layer (such as a speech tokenizer), uses encoder-decoder, decoder-only, etc. architecture, and is trained using massive speech or text data in a self-supervised learning manner. The ASR (speech recognition) system architecture based on the large language model LLM is as shown in Figure 2 Due to the large-scale improvement of training data and model parameters, the LLM-based ASR system has achieved a substantial improvement in overall recognition accuracy, recognition rate in noisy environments, etc., and has strong parameter and data scale expansion capability.

[0004] However, the LLM-based ASR system has the disadvantage of autoregressive decoding for non-streaming recognition, as shown in Figure 3As shown, it is necessary to wait for the user to finish speaking before autoregressive decoding can be performed, resulting in further increase in the delay of human-computer interaction, and the streaming on-screen function cannot be realized (the streaming on-screen function refers to real-time display of recognition results on the screen during the user's speaking process in voice interaction of a device with a screen (such as a car machine, a television, a smart speaker with a screen, and the like)), and therefore is not applicable to interactive scenarios requiring real-time feedback. Meanwhile, when a system is customized for a specific industry, only a large amount of industry voice can be collected and labeled for model training, and the labeling cost and technical threshold are high, the period is long, and the versatility and flexibility are insufficient. SUMMARY

[0005] In view of this, the purpose of the present application is to propose a hybrid speech recognition method and system based on an AI large model architecture, which retains the advantages of traditional ASR field customization, flexible streaming output, low cost, and the like, while simultaneously fusing the powerful modeling capability of an AI large model to improve the overall recognition accuracy; the advantages of flexible customization of a traditional ASR system for a vertical field and streaming recognition are fused with the advantages of an ASR system driven by a large language model (LLM) in terms of recognition accuracy and model capability improvement; a large amount of unlabeled voice data is used for efficient pre-training, and acoustic and language information are more effectively combined, effectively reducing the data labeling cost and technical threshold of system customization; through self-supervised learning, modular architecture, joint training, and multi-output head design, the unification of streaming and high-precision two recognition modes is realized, the online recognition demand of streaming and low latency is supported while ensuring the high-precision recognition capability of an AI large model, the user real-time interaction experience is improved; and through multi-modal fusion of voice and text, user instructions and context information are supported, the field adaptability and user experience are effectively improved, and the method is applicable to various scenarios such as human-computer interaction, voice assistants, real-time transcription, and the like.

[0006] The present application provides a hybrid speech recognition method based on an AI large model architecture, comprising the following steps:

[0007] S1, a hybrid system containing a traditional ASR system (acoustic model + language model + decoder) and an ASR system of an AI large model (a multi-modal voice / text self-supervised model based on a decoder-only or encoder-decoder) is constructed; a double-end output head is arranged in the core model of the hybrid system, one end of the double-end output head is used for directly outputting an acoustic recognition unit (such as CTC / pinyin / word / phoneme), and independently serving as an acoustic model embedded in a traditional ASR system; the other end outputs a text Token sequence, and supports LLM / ASR end-to-end capability;

[0008] S2, in the model training stage, based on a large amount of unlabeled audio data, the AI large model is pre-trained through self-supervised learning, the characteristics of the audio signal are mined, and the AI large model is supervised fine-tuned SFT by using limited high-quality labeled speech data; And introduce human feedback reinforcement learning RLHF to further optimize the recognition accuracy;

[0009] Specifically, human feedback reinforcement learning (RLHF) can realize precise alignment of large language model and human preference.

[0010] S3, the cross entropy loss is used to train the text output in the base layer of the AI large model; The branch of acoustic modeling is introduced in the middle layer or auxiliary head, and the CTC loss function suitable for block alignment is used to train the acoustic output; The acoustic and text information fusion is realized by training the CTC loss function and the cross entropy loss, and the performance of end-to-end and traditional decoding model is improved;

[0011] Cross entropy loss (Cross Entropy Loss) is commonly used to measure the difference between two probability distributions, and is widely used in machine learning, especially in classification problems. In the joint loss and hierarchical output of the present application, the cross entropy loss is used to train the text output in the base layer of the AI large model.

[0012] Specifically, acoustic modeling is a key link in speech recognition system, and acoustic modeling aims to establish the mapping relationship between the acoustic characteristics of speech signal and language information, and convert the speech waveform spoken by human into information that can be understood and processed by computer.

[0013] S4, in the model training or inference stage, the audio block method and KV cache (KV-cache) technology are used to realize efficient streaming recognition; When the audio block input decoding model, the previous activation state is reserved, the recognition result is supported while speaking, the delay is greatly reduced, the user experience is improved, after the speech is finished, the AI large model is based on all cache information, and the subsequent autoregressive text inference is quickly completed, and high-precision recognition text is output.

[0014] Specifically, the audio block method is to divide a piece of audio into multiple segments for acoustic model training, the purpose is to let the acoustic model support streaming speech recognition (user speaks while recognizing, reduce delay).

[0015] In traditional computing, all key-value information needs to be recalculated every time a new token is generated. In the field of speech recognition and NLP, with the expansion of model size and the higher requirement for response speed, KV cache technology emerges as the times require. KV cache is based on the self-attention mechanism of Transformer. During inference, the calculated key-value is stored, and when a new token arrives, only the current key-value is calculated, and the attention calculation is performed after splicing with historical data. KV cache significantly improves the model inference efficiency by optimizing the attention mechanism calculation of the Transformer architecture.

[0016] The working mechanism of KV cache is as follows:

[0017] Initialization: Prepare empty key-value cache according to batch size, maximum content length and information dimension;

[0018] First unit: After the first token is input, calculate the key-value and store it in the corresponding position of the cache;

[0019] Subsequent unit: Only calculate the key-value of the new token, splice with the historical data in the cache to complete the calculation, and update the cache;

[0020] Cache management: When the preset length is reached, update it in a sliding window manner, delete the earliest data to accommodate new content.

[0021] The advantages of KV cache are as follows:

[0022] Reduce the amount of calculation: reuse historical results, significantly reduce the amount of calculation when processing long sequences;

[0023] Improve speed: reduce repeated calculation, reduce delay, suitable for real-time response scenarios;

[0024] Optimize memory: save memory overhead for repeated calculation, achieve overall optimization.

[0025] Preferably, in the field of customization and mixed output, the application supports the fusion of model output and traditional N-gram language model scores, quickly customizes the language model of the field by combining external field text, and combines multiple source scores through the decoder without the need for large amounts of speech data re-labeling, short cycle and low cost, meeting the actual needs of vertical industries;

[0026] Preferably, in the aspect of personalized input and context enhancement, the input layer can introduce user personalized instructions and context information to provide the ability of targeted recognition and context adaptation, and improve the recognition accuracy and intelligent level in complex scenarios.

[0027] Further, the method for supervised fine-tuning of the AI large model in the S2 step comprises the following steps:

[0028] S201, unsupervised pre-training of AI large model on large-scale general data to establish basic feature extraction capability; for example, GPT-3 processes hundreds of millions of text data to master the basic rules of natural language, providing a solid foundation for subsequent optimization.

[0029] S202, build a dedicated annotation dataset around the target task, the data volume of the dedicated annotation dataset is less than the pre-training scale, but it is highly consistent with the application scenario; for example, in the medical field, medical records, diagnosis reports and other texts need to be collected, and structured information such as disease classification and symptom extraction needs to be annotated.

[0030] S203, use the annotated data to train the pre-trained model, and continuously optimize the loss function to adjust the model parameters to the task requirements, and gradually improve the prediction accuracy in specific scenarios;

[0031] S204, use an independent test set that does not participate in training to test the model performance, and quantitatively evaluate it through accuracy and recall rate; if the quantitative evaluation effect does not meet the expectation, improve it by increasing the training data, adjusting the hyperparameters or optimizing the model structure, until the accuracy and recall rate quantitative evaluation effect meets the expectation.

[0032] Supervised Fine-Tuning (SFT) is a core technology to improve the performance of AI models, which realizes the precise upgrade of model capabilities by injecting domain knowledge into pre-trained models. SFT starts with pre-trained models, which have completed the construction of basic capabilities through massive unlabeled data, such as language models that have mastered grammar structures and semantic logic. For specific tasks, high-quality annotated datasets need to be constructed, each sample containing input content and corresponding standard answers. Then, by minimizing the error between the model's prediction results and the labeled labels, optimization algorithms such as gradient descent are used to iteratively update the model parameters, so that the model learns task-specific rules, and finally realizes function customization.

[0033] Further, the method for further optimizing the recognition accuracy by introducing human feedback reinforcement learning RLHF in the S2 step includes the following steps:

[0034] S211, collect human feedback data through a multi-dimensional annotation system, including: star rating of AI large model generated content, and sorting multiple candidate answers according to their merits; the human feedback data collected by the annotation system constitutes the basis corpus for subsequent model training;

[0035] S212, build a reward model, use the collected feedback data as supervised learning samples to train the reward model, and the reward model can quantitatively evaluate the pros and cons of different outputs and learn the internal logic of human judgment of text quality;

[0036] S213, set an initial strategy for the AI large model, make the AI large model interact with the virtual environment and generate output content; score the output result by the reward model and give a corresponding reward value; use the proximal policy optimization (PPO) reinforcement learning algorithm to adjust the strategy according to the reward signal, optimize the content generation logic, and realize iterative update of the strategy optimization:

[0037] S214, after the strategy is updated, continuously collect human feedback data of new output content, retrain the reward model using the new data, and iteratively update the AI large model strategy based on the optimized reward signal, through a closed-loop improvement mechanism, and continuously improve the degree of fit between the model output and human values and use preferences.

[0038] Further, the feature mining of the audio signal in the S2 step includes: mask prediction, frame restoration, and prediction of the next speech token unit.

[0039] The mask prediction method includes: when processing audio data, randomly mask part of the audio segment, and the model predicts the content of the masked part according to the audio information that is not masked; in actual operation, the frequency spectrum of the audio can be processed, and part of the frequency or time segment is randomly selected for masking, and then the model is trained to predict the audio features of these masked areas, such as audio frequency and amplitude information; through a large number of such training, the large model can better understand the structure and features of the audio and improve the processing capability for various audio conditions;

[0040] The frame restoration method includes: dividing the audio into multiple frames according to a certain time interval, and performing special processing on part of the frames, such as adding noise, changing amplitude, etc., to let the model learn to restore these processed frames to the original normal frames; when dividing the audio frames, set appropriate frame length and frame shift according to the characteristics of the audio, then interfere with the selected frames, and the model learns how to remove the interference to restore the original frame features; it can enhance the capture and restoration ability of audio details and improve the recognition performance in complex audio environment;

[0041] The method for predicting the next speech token unit includes: the model predicts the next speech token unit based on the information of the processed speech token unit when processing a piece of audio; the model processes the input audio in turn, and after processing the current token unit, combines the existing context information (including the previous tokenization result, the overall characteristics of the audio, etc.) to predict the next most likely speech token unit. Through a large number of such training, the large model can learn the coherence of speech in semantics and syntax, and improve the understanding and recognition accuracy of speech content.

[0042] Further, the method of step S3 for training the text output using the cross-entropy loss includes:

[0043] For speech recognition that needs to recognize the word text in a segment of speech, the expression of the cross-entropy loss is:

[0044]

[0045] wherein, represents the probability distribution of the real text label, and then is the probability of each possible word (i.e., the recognition result category) appearing in the real situation; for a specific speech segment, there is only one word actually spoken, so the value on the correct word position is 1, and the values on the other positions are 0.

[0046] represents the text probability distribution predicted by the model, and the model will output a probability value for each possible recognition result after a series of calculations. The distribution composed of these probability values is ; for example, the model predicts the text corresponding to a segment of speech, and it will give a probability belonging to each possible word. These probabilities constitute the distribution;

[0047] n represents the number of all possible categories, which is the total number of words in the vocabulary in the speech recognition scenario, i.e., the number of all recognition results that the model can output; if there are 1000 words in the vocabulary, then n = 1000;

[0048] By calculating the cross-entropy loss between the predicted text probability distribution and the real text label probability distribution , the model continuously adjusts its parameters so that the prediction result is as close to the real situation as possible. The goal of model training is to minimize this cross-entropy loss value, so that the prediction distribution is more consistent with the real distribution, thereby improving the accuracy of the speech recognition system in recognizing text.

[0049] Further, the method of step S3 for training the acoustic output using the CTC loss function suitable for block alignment includes:

[0050] The CTC loss function formula is constructed as:

[0051]

[0052] wherein, the probability of the output sequence is calculated by the forward-backward algorithm, i.e., given the input sequence , the probability of the output sequence is calculated.

[0053]

[0054] wherein, is the total number of training samples, such as 1000 training speech segments in speech recognition, covering different sentences, speakers and environmental noise;

[0055] is the th input sample, which is the sequence of preprocessed and feature extracted speech, such as MFCC, FBANK, etc. feature sequence;

[0056] is the corresponding true label sequence, i.e. the text content actually expressed by the speech;

[0057] is the intermediate sequence predicted by the model, which is longer and contains blank symbols (“-”), solving the problem of speech text alignment;

[0058] is the probability of given , reflecting the prediction possibility;

[0059] is the probability of given , which is obtained by summing the probabilities of related intermediate sequences, and the model improves the recognition accuracy by minimizing the loss function .

[0060] Specifically, Connectionist Temporal Classification (CTC) is an algorithm for processing sequence-to-sequence problems, and in the scheme of introducing a special acoustic modeling branch in the intermediate layer or auxiliary head of the present application, the CTC loss function can make the newly added branch handle the complex relationship between speech and text, and improve the performance of the speech recognition system.

[0061] Further, the method of introducing an acoustic modeling branch in the S3 step includes:

[0062] analyzing and modeling the acoustic characteristics of the speech, cutting the speech signal into a plurality of shorter frames in chronological order, and extracting feature parameters of each frame of speech, wherein the feature parameters include Mel Frequency Cepstral Coefficients (MFCC) and Linear Predictive Cepstral Coefficients (LPCC), and the feature parameters can reflect the acoustic characteristics of the speech;

[0063] The acoustic model learns the acoustic performance rules of different speech units (such as phonemes, syllables, etc.) in various pronunciation conditions based on the feature parameters through a statistical method or a machine learning algorithm.

[0064] The application also provides a hybrid speech recognition system based on an AI large model architecture, which executes the hybrid speech recognition method based on the AI large model architecture as described above, and comprises:

[0065] The model architecture fusion module is used to construct a hybrid system of the traditional ASR system and the ASR system containing the AI large model; a double-end output head is arranged in the core model of the hybrid system, one end of the double-end output head is used to directly output an acoustic recognition unit and is independently embedded into the traditional ASR system as an acoustic model, and the other end outputs a text Token sequence, supporting LLM / ASR end-to-end capability;

[0066] The multi-modal self-supervised training module is used to pre-train the AI large model through a self-supervised learning method based on a large amount of unlabeled audio data in the model training stage, mine features of the audio signal, and fine-tune the model SFT in a supervised manner by using limited high-quality labeled speech data; and human feedback reinforcement learning RLHF is introduced to further optimize the recognition accuracy;

[0067] The joint loss and layered output module is used to train the text output by using a cross-entropy loss in the base layer of the AI large model; an acoustic modeling branch is introduced in the middle layer or auxiliary head, and a CTC loss function suitable for block alignment is used to train acoustic output; the CTC loss function and the cross-entropy loss are combined to train the acoustic and text information fusion;

[0068] The streaming inference and KV cache module is used to realize efficient streaming recognition by using an audio block method and a KV cache technology in the model training or inference stage: when the audio block is input into the decoding model, the previous activation state is retained, the scheme supports saying and outputting the recognition result at the same time, greatly reduces the delay, and improves the user experience. After the speech is finished, the model can quickly complete the subsequent autoregressive text inference based on all the cached information, and output high-precision recognition text.

[0069] The application adopts a dual output / mixed architecture, realizes compatibility of high precision and streaming experience, can play the advantages of high end-to-end recognition rate and strong semantic understanding of an AI large model, and retains the ability of a traditional ASR system in streaming inference, real-time feedback and field flexible customization; the pre-training is performed by using a large amount of unlabeled voice, the dependence on labeled data is greatly reduced, the industry customization threshold and cost are reduced, and this is beneficial to accelerate industry scene adaptation and landing; the innovative multi-output head architecture and loss joint training improve the information coupling and generalization ability of the model, effectively overcome the model mismatch and cumulative error problems of the traditional scheme, the KV cache and block inference are adopted, the model is freely switched between display while speaking, low-delay streaming voice recognition and high-precision post-processing, and the user experience is greatly improved; the dual-track output and multi-modal access are supported, linkage expansion with other downstream AI tasks (such as semantic understanding, intelligent question answering and the like) is facilitated, and the application space is extended; and the field optimization cycle of the application scheme is short, a large amount of manual labeling is not needed, and the project delivery efficiency is improved.

[0070] The application further provides a computer readable storage medium, which stores a computer program, and the program realizes the steps of the mixed voice recognition method based on the AI large model architecture when executed by a processor.

[0071] The application further provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor realizes the steps of the mixed voice recognition method based on the AI large model architecture when executing the program.

[0072] Compared with the prior art, the application has the following beneficial effects:

[0073] The hybrid speech recognition method and system based on the AI large model architecture provided by the application retain the advantages of traditional ASR field customization, flexible streaming output, and low cost, while combining the powerful modeling capability of the AI large model to improve the overall recognition accuracy. The advantages of traditional ASR system in flexible customization and streaming recognition for vertical fields are combined with the advantages of LLM-driven ASR system in recognition accuracy and model capability improvement. The massive unlabeled speech data is used for efficient pre-training, and the acoustic and language information is more effectively combined, effectively reducing the data labeling cost and technical threshold of system customization. Through self-supervised learning, modular architecture, joint training, and multi-output head design, the unified recognition mode of streaming and high-precision is realized, which guarantees the high-precision recognition capability of the AI large model, supports streaming and low-latency online recognition requirements, and improves the user real-time interaction experience. Through the multi-modal fusion of voice and text, user instructions and context information are added to effectively improve the field adaptability and user experience, which is suitable for various scenes such as human-computer interaction, voice assistant, and real-time transcription, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0074] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustration only and are not considered a limitation of the application.

[0075] In the drawings:

[0076] Figure 1 is a traditional hybrid speech recognition system framework diagram;

[0077] Figure 2 is an existing LLM-based ASR system architecture diagram;

[0078] Figure 3 is an existing LLM-based ASR autoregressive decoding example diagram;

[0079] Figure 4 is a self-supervised learning network diagram based on the AI large model architecture of the embodiment of the application;

[0080] Figure 5 is an architecture diagram of the hybrid speech recognition system based on the AI large model architecture of the embodiment of the application;

[0081] Figure 6 is a flowchart of a hybrid speech recognition method based on the AI large model architecture of the application;

[0082] Figure 7 is a schematic diagram of the computer device according to the embodiment of the application. DETAILED DESCRIPTION

[0083] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are not representative of all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and articles of manufacture consistent with some aspects of the present disclosure as detailed in the appended claims.

[0084] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0085] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one from another. For example, a first information can be termed a second information, and, similarly, a second information can also be termed a first information, without departing from the scope of the present disclosure. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0086] The embodiments of the present application are further described below.

[0087] The embodiments of the present application provide a hybrid speech recognition method based on an AI large model architecture, referring to Figure 6 as shown, comprising the following steps:

[0088] S1, a hybrid system of an ASR system containing a traditional ASR system and an AI large model is constructed; a double-end output head is arranged in the core model of the hybrid system, one end of the double-end output head is used to directly output an acoustic recognition unit and is independently embedded into the traditional ASR system as an acoustic model; the other end outputs a text Token sequence, supporting LLM / ASR end-to-end capability;

[0089] S2, in the model training stage, based on a large amount of unlabeled audio data, the AI large model is pre-trained through a self-supervised learning method, the features of the audio signal are mined, and the AI large model is supervised fine-tuned SFT using limited high-quality labeled speech data; and human feedback reinforcement learning RLHF is introduced to further optimize the recognition accuracy;

[0090] The neural network architecture corresponding to the model training stage is as shown in Figure 4 ;

[0091] The method of supervised fine-tuning of the AI large model comprises the following steps:

[0092] S201, unsupervised pre-training of the AI large model on large-scale general data to establish basic feature extraction capability; for example, GPT-3 processes hundreds of millions of text data to master the basic rules of natural language, providing a solid foundation for subsequent optimization.

[0093] S202, build a dedicated annotation dataset around the target task, the data volume of the dedicated annotation dataset is smaller than the pre-training scale, but highly fits the application scenario; for example, the medical field needs to collect medical records, diagnosis reports and other texts, and annotate structured information such as disease classification and symptom extraction.

[0094] S203, use the annotated data to train the pre-trained model, and iteratively optimize the loss function to adjust the model parameters to the task requirements, gradually improve the prediction accuracy in specific scenarios;

[0095] S204, use an independent test set that does not participate in training to test the model performance, and quantitatively evaluate it through accuracy and recall rate indicators; if the quantitative evaluation effect does not meet the expectation, improve it by increasing the training data, adjusting the hyperparameters or optimizing the model structure, until the accuracy and recall rate indicators meet the expectation.

[0096] The method of introducing human feedback reinforcement learning RLHF to further optimize the recognition accuracy comprises the following steps:

[0097] S211, collect human feedback data through a multi-dimensional annotation system, including: star rating of the AI large model generated content, and sorting multiple candidate answers according to their merits; the human feedback data collected by the annotation system constitutes the basis corpus for subsequent model training;

[0098] S212, build a reward model, use the collected feedback data as a supervised learning sample to train the reward model, the reward model can quantitatively evaluate the pros and cons of different outputs, and learn the internal logic of human judgment of text quality;

[0099] S213, set an initial strategy for the AI large model, make the AI large model interact with the virtual environment and generate output content; score the output results by the reward model and give corresponding reward values; use the proximal policy optimization (PPO) reinforcement learning algorithm to adjust the strategy according to the reward signal, optimize the content generation logic, and realize the iterative update of the strategy optimization:

[0100] S214, after the policy is updated, the human feedback data of the new output content is continuously collected, the reward model is retrained using the new data, and the AI large model policy is iterated again based on the optimized reward signal, iterated and optimized through the closed-loop cycle improvement mechanism, and the consistency of the model output with the human values and use preferences is continuously improved.

[0101] The feature mining of the audio signal includes: mask prediction, frame restoration, and prediction of the next speech token unit.

[0102] The mask prediction method includes: when processing audio data, randomly masking part of the audio segment, and the model predicting the content of the masked part according to the audio information that is not masked; in actual operation, the spectrum of the audio can be processed, and part of the frequency or time segment is randomly selected for masking, and then the model is trained to predict the audio features of these masked areas, such as the frequency and amplitude information of the audio; through a large number of such training, the model can better understand the structure and features of the audio, and improve the processing capability of various audio situations;

[0103] The frame restoration method includes: dividing the audio into multiple frames according to a certain time interval, and specially processing part of the frames, including adding noise, changing amplitude, etc., so that the model learns to restore these processed frames to the original normal frames; when dividing the audio frames, set appropriate frame length and frame shift according to the characteristics of the audio, then interfere with the selected frames, and the model learns how to remove the interference to restore the features of the original frames; enhance the ability to capture and restore the details of the audio, and improve the recognition performance in complex audio environment;

[0104] The method of predicting the next speech token unit includes: the model predicts the next speech token unit according to the information of the processed speech token unit when processing a piece of audio; the model processes the input audio in turn, and after processing the current token unit, it combines the existing context information, including the previous tokenization result, the overall characteristics of the audio, etc., to predict the next most likely speech token unit. Through a large number of such training, the model can learn the coherence of speech in semantics and syntax, and improve the understanding and recognition accuracy of speech content.

[0105] S3, train the text output using cross-entropy loss at the base layer of the AI large model; introduce an acoustic modeling branch in the middle layer or auxiliary head, use a CTC loss function suitable for block alignment to train acoustic output; jointly train the CTC loss function and the cross-entropy loss to realize the fusion of acoustic and text information, and improve the performance of end-to-end and traditional decoding models;

[0106] The method for training the text output using cross-entropy loss includes:

[0107] For speech recognition, the cross-entropy loss is expressed as:

[0108]

[0109] where represents the probability distribution of the true text label, and is the probability of each possible word (i.e., the recognition result category) appearing in the true situation; for a specific speech segment, there is only one word actually spoken, so the value at the correct word position is 1, and the rest are 0 (this is a common one-hot encoding representation of the true label). For example, if the correct recognition result is "apple", the probability of "apple" corresponding to the position in the probability distribution vector of all possible words is 1, and the probability of other word positions is 0.

[0110] represents the text probability distribution predicted by the model, in this hybrid speech recognition system, the model will output a probability value for each possible recognition result after a series of calculations, and the distribution composed of these probability values is ; for example, the model predicts the text corresponding to a speech segment, and it will give a probability belonging to each possible word, and these probabilities form the distribution;

[0111] n represents the number of all possible categories, in the speech recognition scenario, it is the total number of words in the vocabulary, i.e., the number of all recognition results that the model can output; if there are 1000 words in the vocabulary, then n = 1000;

[0112] By calculating the cross-entropy loss between the predicted text probability distribution and the true text label probability distribution , the model continuously adjusts its parameters to make the prediction as close to the true situation as possible, so that the prediction distribution is more consistent with the true distribution, improving the accuracy of the speech recognition system in recognizing text.

[0113] The method for training the acoustic output using the CTC loss function suitable for block alignment includes:

[0114] The CTC loss function formula is constructed as:

[0115]

[0116] where the probability of the output sequence is calculated by the forward-backward algorithm, which is: specifically:

[0117]

[0118] wherein, is the total number of training samples, such as 1000 training speech segments in speech recognition, covering different sentences, speakers and environmental noise;

[0119] is the th input sample, which is the sequence of pre-processed and feature-extracted speech, such as MFCC, FBANK, etc.

[0120] is the corresponding true label sequence, i.e. the text content actually expressed by the speech;

[0121] is the intermediate sequence predicted by the model, which is longer and contains blank symbols (“-”), solving the problem of speech text alignment;

[0122] is the probability of given , reflecting the prediction possibility;

[0123] is the probability of obtaining the true label given , which is obtained by summing the probabilities of related intermediate sequences, and the model improves the recognition accuracy by minimizing the loss function .

[0124] The embodiment introduces a special acoustic modeling branch in the intermediate layer or auxiliary head, and the CTC loss function enables the newly added branch to handle the complex relationship between speech and text, thereby improving the performance of the speech recognition system.

[0125] The method of introducing the branch of acoustic modeling includes:

[0126] The acoustic characteristics of the speech are analyzed and modeled, and the speech signal is cut into a plurality of short frames in time sequence, and feature parameters of each frame of speech are extracted, the feature parameters including: mel frequency cepstral coefficient (MFCC), linear predictive cepstral coefficient (LPCC), which can reflect the acoustic characteristics of the speech;

[0127] The acoustic model learns the acoustic performance rules of different speech units (including phonemes, syllables, etc.) in various pronunciation conditions based on the feature parameters through statistical methods or machine learning algorithms.

[0128] When identifying the word "apple", acoustic modeling analyzes the acoustic characteristics of each phoneme that makes up the pronunciation of "apple" in the actual speech, such as the differences in speech waveform and acoustic characteristics when different people pronounce "apple" due to pronunciation habits, speech speed, and intonation. The acoustic model captures these change patterns and establishes a reliable mapping relationship to enable the system to accurately convert speech into corresponding text. In the "introducing a special acoustic modeling branch in the middle or auxiliary head" you mentioned, a part that focuses on learning and processing speech acoustic features is added to the model architecture, further improving the speech recognition system's processing ability and recognition accuracy of speech signals.

[0129] S4, in the model training or inference stage, the audio blocking method and the KV cache technology are adopted to realize efficient streaming recognition; when the audio blocking input decoding model, the previous activation state is reserved, the recognition result is supported while speaking, and after the speech is finished, the AI large model is based on all the cache information, and the subsequent autoregressive text inference is quickly completed, and the high-precision recognition text is output.

[0130] The audio blocking method divides a piece of audio into multiple segments for acoustic model training, aiming to enable the acoustic model to support streaming speech recognition (recognize while speaking, reduce latency).

[0131] In traditional computing, all key-value information needs to be recalculated every time a new token is generated. KV cache technology stores the calculated key-value during inference, and only calculates the current key-value when a new token arrives. After splicing with historical data, attention calculation is performed. KV cache significantly improves model inference efficiency by optimizing the attention mechanism calculation of the Transformer architecture.

[0132] The embodiment of the application also provides a hybrid speech recognition system based on an AI large model architecture, which executes the hybrid speech recognition method based on the AI large model architecture as described above, comprising:

[0133] The model architecture fusion module is used to build a hybrid system containing a traditional ASR system and an AI large model ASR system. A double-end output head is arranged in the core model of the hybrid system. One end of the double-end output head is used to directly output an acoustic recognition unit and is independently embedded as an acoustic model in the traditional ASR system. The other end outputs a text token sequence, supporting LLM / ASR end-to-end capability.

[0134] The multi-modal self-supervised training module is used to pre-train the AI large model based on a large amount of unlabeled audio data through a self-supervised learning method during the model training stage, to mine the features of the audio signal, to use limited high-quality labeled speech data to supervise the fine-tuning SFT of the model, and to introduce human feedback reinforcement learning RLHF to further optimize the recognition accuracy.

[0135] Joint loss and hierarchical output module: for training text output at the base layer of the AI large model using cross-entropy loss; introducing a branch of acoustic modeling at the middle layer or auxiliary head, training acoustic output using a CTC loss function suitable for block alignment; jointly training the CTC loss function and the cross-entropy loss to realize acoustic and text information fusion;

[0136] Streaming inference and KV cache module: for efficient streaming recognition in the model training or inference stage, using audio block input and KV cache technology: when the audio block is input into the decoding model, the previous activation state is retained, the scheme supports real-time recognition results while speaking, greatly reduces the delay, and improves the user experience. After the speech is finished, the model can quickly complete the subsequent autoregressive text inference based on all the cached information, and output high-precision recognition text.

[0137] Figure 5 The mixed speech recognition system architecture based on the AI large model architecture of the embodiment is shown.

[0138] The embodiment of the application also provides a computer device, Figure 7 is a structural schematic diagram of a computer device provided by the embodiment of the application; referring to the accompanying drawings Figure 7 As shown in the figure, the computer device comprises an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used for storing one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 realize the mixed speech recognition method based on the AI large model architecture provided by the above-mentioned embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected through a bus or other means, Figure 7 for example, by bus connection.

[0139] The memory 22 is a kind of readable and writable storage medium of computing device, which can be used to store software programs, computer executable programs, such as the program instructions of the mixed speech recognition method based on the AI large model architecture described in the embodiment of the application; the memory 22 can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system and at least one application program required by a function; the storage data area can store data created according to the use of the device and the like; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state storage device; in some examples, the memory 22 can further include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0140] The input system 23 can be used to receive inputted digital or character information, and to generate key signal inputs related to user settings and function controls of the device; the output system 24 can include a display device such as a display screen.

[0141] The processor 21 performs various function applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, i.e. implements the hybrid speech recognition method based on the AI large model architecture as described above.

[0142] The computer device provided above can be used to perform the hybrid speech recognition method based on the AI large model architecture provided in the above embodiments, and has the corresponding functions and advantages.

[0143] The embodiments of the present application also provide a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform the hybrid speech recognition method based on the AI large model architecture provided in the above embodiments. The storage medium is any various types of memory devices or storage devices, and the storage medium includes: installation media such as CD-ROM, floppy disk or tape system; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as hard disk or optical storage); register or other similar types of memory elements, etc.; the storage medium can also include other types of memory or combinations thereof; in addition, the storage medium can be located in the first computer system in which the program is executed, or can be located in a different second computer system, which is connected to the first computer system through a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected through a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) executable by one or more processors.

[0144] Of course, the storage medium containing computer executable instructions provided by the embodiments of the present application is not limited to the hybrid speech recognition method based on the AI large model architecture as described in the above embodiments, but can also perform the related operations in the hybrid speech recognition method based on the AI large model architecture provided by any embodiments of the present application.

[0145] So far, the technical solutions of the present application have been described in combination with the preferred embodiments, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will all fall within the protection scope of the present application.

[0146] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application; the present application can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A hybrid speech recognition method based on an AI large-scale model architecture, characterized in that, Includes the following steps: S1. Construct a hybrid ASR system that includes a traditional ASR system and an AI large model; set a dual-end output head in the core model of the hybrid system. One end of the dual-end output head is used to directly output the acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs a text token sequence, supporting LLM / ASR end-to-end capabilities. S2. During the model training phase, the AI ​​model is pre-trained using a self-supervised learning method based on a large amount of unlabeled audio data to mine the features of audio signals. The AI ​​model is then fine-tuned using supervised SFT with limited high-quality labeled speech data. Furthermore, human feedback reinforcement learning (RLHF) is introduced to further optimize the recognition accuracy. S3. Train the text output using cross-entropy loss in the base layer of the AI ​​model; introduce an acoustic modeling branch in the intermediate layer or auxiliary head, and train the acoustic output using a CTC loss function suitable for block alignment. By combining the CTC loss function with the cross-entropy loss during training, acoustic and textual information fusion is achieved, while simultaneously improving the performance of end-to-end and traditional decoding models. S4. During the model training or inference phase, audio segmentation and KV caching technology are used to achieve efficient streaming recognition. When audio segments are input into the decoding model, the previous activation state is retained, supporting recognition results while speaking. After speaking, the AI ​​big model quickly completes subsequent autoregressive text inference based on all cached information and outputs high-precision recognized text.

2. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The supervised fine-tuning method for the large AI model in step S2 includes: S201. Perform unsupervised pre-training on large-scale general data to establish basic feature extraction capabilities for large AI models. S202. Construct a dedicated labeled dataset around the target task. The data volume of the dedicated labeled dataset is smaller than the pre-training scale, but it is highly consistent with the application scenario. S203. Use labeled data to train the pre-trained model in a targeted manner, and continuously optimize the loss function to adjust the model parameters in line with the task requirements, thereby gradually improving the prediction accuracy in specific scenarios. S204. Use an independent test set that was not used in training to test the model performance, and evaluate it quantitatively using accuracy and recall metrics. If the quantitative evaluation effect does not meet expectations, improve it by increasing training data, adjusting hyperparameters, or optimizing the model structure until the quantitative evaluation effect of accuracy and recall metrics meets expectations.

3. The hybrid speech recognition method based on an AI large model architecture according to claim 2, characterized in that, The method for further optimizing recognition accuracy by introducing human feedback reinforcement learning (RLHF) in step S2 includes: S211. Collect human feedback data through a multi-dimensional annotation system, including: giving star ratings to the content generated by the AI ​​large model and ranking multiple candidate answers according to their quality; the human feedback data collected by the annotation system constitutes the basic corpus for subsequent model training. S212. Construct a reward model, use the collected feedback data as supervised learning samples to train the reward model, the reward model can quantitatively evaluate the quality of different outputs and learn the inherent logic of human judgment of text quality. S213. Set an initial strategy for the large AI model, enabling it to interact with the virtual environment and generate output content; the reward model scores the output results and assigns corresponding reward values; the reinforcement learning algorithm is optimized using a proximal policy, adjusting its own strategy based on reward signals, optimizing the content generation logic, and achieving iterative updates to the strategy optimization: S214. After the strategy is updated, human feedback data of the new output content is continuously collected, the reward model is retrained using the new data, and the AI ​​large model strategy is iterated again based on the optimized reward signal. Through a closed-loop cyclic improvement mechanism, the model output is continuously improved to better match human values ​​and usage preferences.

4. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The features of the audio signal mined in step S2 include: masking prediction, frame restoration, and prediction of the next speech segmentation unit; The masking prediction method includes: when processing audio data, randomly masking some audio segments, and the model predicting the content of the masked parts based on the unmasked audio information; in actual operation, the audio spectrogram can be processed, some frequencies or time segments can be randomly selected for masking, and then the model can learn to predict the audio features of these masked areas, such as the frequency and amplitude information of the audio. The frame restoration method includes: dividing the audio into multiple frames according to a certain time interval, performing special processing on some frames, and allowing the model to learn to restore these processed frames to the original normal frames; when dividing the audio frames, setting an appropriate frame length and frame shift according to the characteristics of the audio, and then performing interference operations on the selected frames. The model learns continuously how to remove these interferences and restore the features of the original frames. The method for predicting the next speech segmentation unit includes: when the model processes an audio segment, it predicts the next speech segmentation unit based on the information of the speech segmentation units that have already been processed; the model performs segmentation processing on the input audio segment sequentially, and after processing the current segmentation unit, it predicts the next most likely speech segmentation unit by combining the existing context information.

5. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for training text output using cross-entropy loss in step S3 includes: For speech recognition, which requires identifying words within a speech segment, the expression for the cross-entropy loss is: in, The probability distribution representing the real text labels. It refers to the probability of each possible word appearing in a real situation. The value is 1 at the correct word position and 0 at other positions; This represents the probability distribution of the text predicted by the model. After a series of calculations, the model outputs a probability value for each possible recognition result, and the distribution formed by these probability values ​​is the probability distribution. ; n represents the number of all possible categories, which in the context of speech recognition is the total number of words in the vocabulary, i.e., the number of all possible recognition results that the model can output; By calculating the predicted text probability distribution Probability distribution of real text labels The model continuously adjusts its parameters to minimize the cross-entropy loss between predictions, making the predictions as close to the actual situation as possible.

6. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for training the acoustic output using a CTC loss function suitable for block alignment in step S3 includes: Constructing the CTC loss function: in, Calculated using a forward-backward algorithm, i.e., given an input sequence The output sequence is obtained. The probability is as follows: in, This refers to the total number of training samples, such as 1000 training speech segments in speech recognition, covering different sentences, speakers, and environmental noise; For the first Each input sample is a sequence of speech after preprocessing and feature extraction, such as MFCC, FBANK and other feature sequences; for The corresponding real label sequence, that is, the text content actually expressed by the speech; The intermediate sequence predicted by the model is longer and contains whitespace symbols ("-"), which solves the problem of speech-text alignment. For a given At that time, the model predicts The probability reflects the likelihood of prediction; For a given At that time, the real label is obtained. The probability is obtained by summing the probabilities of the relevant intermediate sequences. The model minimizes the loss function. Improve recognition accuracy.

7. The hybrid speech recognition method based on an AI large model architecture according to claim 1, characterized in that, The method for introducing the acoustic modeling branch in step S3 includes: The acoustic characteristics of speech are analyzed and modeled. The speech signal is divided into short frames in time sequence, and the feature parameters of each frame are extracted. The feature parameters include Mel frequency cepstral coefficients and linear prediction cepstral coefficients. The feature parameters can reflect the acoustic characteristics of speech. The acoustic model, based on the feature parameters, learns the acoustic performance patterns of different speech units under various pronunciation conditions through statistical methods or machine learning algorithms.

8. A hybrid speech recognition system based on an AI large-scale model architecture, characterized in that, Performing the hybrid speech recognition method based on an AI large model architecture as described in any one of claims 1-7, comprising: Model architecture fusion module: used to build a hybrid system of ASR system including traditional ASR system and AI large model; a dual-end output head is set in the core model of the hybrid system. One end of the dual-end output head is used to directly output acoustic recognition unit, which is independently embedded as an acoustic model in the traditional ASR system; the other end outputs text token sequence, supporting LLM / ASR end-to-end capability; Multimodal self-supervised training module: During the model training phase, it is used to pre-train the AI ​​model based on a large amount of unlabeled audio data through self-supervised learning, mine the features of audio signals, and perform supervised fine-tuning of the model using limited high-quality labeled speech data; and introduces human feedback reinforcement learning (RLHF) to further optimize the recognition accuracy. Joint Loss and Hierarchical Output Module: Used to train text output using cross-entropy loss in the base layer of the AI ​​model; introduces acoustic modeling branches in intermediate layers or auxiliary heads, and trains acoustic output using a CTC loss function suitable for block alignment; combines the CTC loss function with the cross-entropy loss to achieve acoustic and text information fusion. Streaming Inference and KV Caching Module: Used during model training or inference phases, this module employs audio segmentation and KV caching technology to achieve efficient streaming recognition. When audio segments are input into the decoding model, the preceding activation states are retained. The solution supports simultaneous speaking and output of recognition results, greatly reducing latency and improving user experience. After speaking, the model can quickly complete subsequent autoregressive text inference based on all cached information, outputting high-precision recognized text.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the hybrid speech recognition method based on the AI ​​large model architecture as described in any one of claims 1-7.

10. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the hybrid speech recognition method based on an AI large model architecture as described in any one of claims 1-7.

Images