A method, system and terminal for continuous speech recognition based on a large language model

Abstract

This invention discloses a continuous speech recognition method, system, and terminal based on a large language model. The method includes: acquiring a speech signal, performing frame-by-frame preprocessing and feature extraction on the speech signal to obtain short-time energy and zero-crossing rate; performing dynamic silence detection based on the short-time energy and zero-crossing rate to obtain a trigger audio stream; determining an end-to-end speech recognition model and transcribing the trigger audio stream using the end-to-end speech recognition model to obtain an original text sequence; determining a large language model and performing semantic error correction and context optimization on the original text sequence using the large language model to obtain a recognized text sequence; and reprocessing the recognized text sequence using a caching and paragraph fusion mechanism to obtain a final speech recognition result. This invention effectively improves the recognition accuracy and semantic coherence of the speech recognition result by deeply integrating the end-to-end speech recognition model with the large language model.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a continuous speech recognition method, system and terminal based on a large language model. Background Technology

[0002] With the development of smart terminals and artificial intelligence technologies, speech recognition has become an important means of human-computer interaction. Existing speech recognition systems typically rely on joint modeling of acoustic and language models to convert speech signals into text sequences. However, in scenarios involving long speech or multi-turn dialogues, these methods are prone to problems such as loss of contextual information, inaccurate segmentation, and unclear understanding of ambiguous words. Furthermore, existing speech recognition systems have limited utilization of speech features such as speech rate, emotion, and accent, making it difficult to achieve personalized recognition.

[0003] With the rise of large language models, they have shown significant advantages in context understanding and semantic reasoning. However, most current speech recognition systems only use large language models in the post-processing stage, failing to achieve deep integration with the speech recognition module. This results in poor performance in terms of real-time performance and context consistency, failing to meet user needs.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a continuous speech recognition method, system, and terminal based on a large language model, aiming to solve the problem that current speech recognition systems in the prior art cannot achieve deep integration of the large language model and the speech recognition module, resulting in poor performance in terms of real-time performance and contextual consistency.

[0006] To achieve the above objectives, the present invention provides a continuous speech recognition method based on a large language model, the method comprising the following steps: Acquire speech signals, and perform frame-by-frame preprocessing and feature extraction on the speech signals to obtain short-time energy and zero-crossing rate; Dynamic silence detection is performed based on the short-time energy and the zero-crossing rate to obtain the trigger audio stream; An end-to-end speech recognition model is determined, and the trigger audio stream is transcribed into text using the end-to-end speech recognition model to obtain the original text sequence. A large language model is determined, and semantic error correction and context optimization are performed on the original text sequence using the large language model to obtain a recognized text sequence; The recognized text sequence is reprocessed using a caching and paragraph fusion mechanism to obtain the final speech recognition result.

[0007] Optionally, the continuous speech recognition method based on a large language model, wherein acquiring the speech signal and performing frame-by-frame preprocessing and feature extraction on the speech signal to obtain short-time energy and zero-crossing rate specifically includes: Real-time acquisition of audio signals, and segmentation of the audio signals into frames according to a preset frame length to obtain framed audio signals; The discrete sample values ​​of the framed speech signal and the number of sample points in each frame are obtained. The short-time energy of each frame is calculated based on the number of sample points and the framed speech signal. The zero-crossing rate of each frame is calculated based on the number of sample points, the discrete sample values, and the framed speech signal. The calculation expression for the short-time energy is as follows: ; in, For the first The short-time energy of a frame, The number of sampling points per frame. Let x(n) be the number of frames, and x(n) be the framed audio signal. The formula for calculating the zero-crossing rate is: ; in, Let n be the zero-crossing rate of the nth frame. For symbolic functions, For the speech signal in the first Discrete sampled values ​​of a frame.

[0008] Optionally, the continuous speech recognition method based on a large language model, wherein the step of performing dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain the trigger audio stream specifically includes: A dynamic silence detection algorithm is used to perform dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a binary silence mask; An audio stream is constructed based on the binary silence mask, the short-time energy, the zero-crossing rate, and the framed speech signal, and the trigger audio stream is extracted from the audio stream.

[0009] Optionally, the continuous speech recognition method based on a large language model, wherein the dynamic silence detection algorithm performs dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a binary silence mask, specifically includes: An adaptive thresholding strategy is used to determine the short-time energy threshold and the zero-crossing rate threshold, as well as the shortest speech duration and the maximum silence duration. Based on the shortest speech duration and the maximum silence duration, the short-time energy is compared with the short-time energy threshold to obtain a first comparison result, and the zero-crossing rate is compared with the zero-crossing rate threshold to obtain a second comparison result; Based on the first comparison result and the second comparison result, a speech frame and a silence frame are obtained, and a binary silence mask is constructed based on the speech frame and the silence frame. The expression for the binary mute mask is as follows: ; in, To dynamically detect the status, This refers to the detection status of the previous frame. For short-time energy threshold, The zero-crossing rate threshold, This represents the length of the currently continuously detected speech. The shortest speech duration, This represents the current continuously detected length of silence. This is the maximum silent duration.

[0010] Optionally, in the continuous speech recognition method based on a large language model, the end-to-end speech recognition model includes an encoder and a decoder. The process of determining an end-to-end speech recognition model and then using that model to transcribe the triggered audio stream into text to obtain the original text sequence specifically includes: Acoustic features are extracted from the triggered audio stream to obtain a frame-level acoustic feature sequence; An end-to-end speech recognition model is determined, and the frame-level acoustic feature sequence is input into the encoder in the end-to-end speech recognition model. The encoder extracts contextual speech features from the frame-level acoustic feature sequence to obtain a latent space representation. The latent space representation is input into the decoder in the end-to-end speech recognition model. The decoder generates a symbol sequence step by step based on the latent space representation to obtain the original text sequence.

[0011] Optionally, the continuous speech recognition method based on a large language model, wherein determining the large language model and performing semantic error correction and context optimization on the original text sequence using the large language model to obtain the recognized text sequence specifically includes: Determine a large language model and input the original text sequence into the large language model; Determine the contextual semantic information, and calculate the sequence maximization generation probability using the large language model based on the original text sequence and the contextual semantic information to obtain the conditional probability; Obtain the confidence probability corresponding to the original text sequence, and calculate the comprehensive error correction score based on the confidence probability and the conditional probability; The formula for calculating the comprehensive error correction score is as follows: ; in, To calculate the overall error correction score, and All are weighted coefficients. Let be the confidence probability. The conditional probability; The text sequence with the highest score in the original text sequence is selected based on the comprehensive error correction score to obtain the recognized text sequence.

[0012] Optionally, the continuous speech recognition method based on a large language model, wherein determining the contextual semantic information involves calculating the conditional probability by maximizing the sequence generation probability based on the original text sequence and the contextual semantic information using the large language model, and then further includes: The context semantic information is dynamically updated to obtain updated context semantic information; The expression for the dynamic update is: ; in, For contextual semantic information, The context state of the previous round, For the text sequence identified in the previous round, Encode the emotion or speech features from the previous round. This is a context fusion function.

[0013] Optionally, the continuous speech recognition method based on a large language model, wherein the reprocessing of the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result specifically includes: The recognized text sequence is generated in multiple rounds based on the updated context semantic information to obtain multiple rounds of recognition results; The results of the multiple rounds of recognition are merged in chronological or contextual order to obtain a continuous text stream; The continuous text stream is cleaned, standardized, semantically optimized, and contextually optimized to obtain the final speech recognition result.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a continuous speech recognition system based on a large language model, wherein the continuous speech recognition system based on a large language model includes: The feature extraction module is used to acquire the speech signal and perform frame-by-frame preprocessing and feature extraction on the speech signal to obtain the short-time energy and zero-crossing rate. A dynamic silence detection module is used to perform dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a trigger audio stream; The text transcription module is used to determine the end-to-end speech recognition model and perform text transcription on the triggered audio stream through the end-to-end speech recognition model to obtain the original text sequence. The text sequence optimization module is used to determine a large language model and perform semantic error correction and context optimization on the original text sequence through the large language model to obtain a recognized text sequence. The speech recognition result output module is used to reprocess the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result.

[0015] In this invention, a speech signal is acquired, and the speech signal is preprocessed by frame segmentation and feature extraction to obtain short-time energy and zero-crossing rate. Dynamic silence detection is performed based on the short-time energy and zero-crossing rate to obtain a trigger audio stream. An end-to-end speech recognition model is determined, and the trigger audio stream is transcribed into text using the end-to-end speech recognition model to obtain an original text sequence. A large language model is determined, and the original text sequence is semantically corrected and context-optimized using the large language model to obtain a recognized text sequence. A caching and paragraph fusion mechanism is used to further process the recognized text sequence to obtain the final speech recognition result. This invention, through feature extraction and dynamic silence detection of the speech signal, can accurately extract the trigger audio stream. Furthermore, the deep fusion of the end-to-end speech recognition model and the large language model enables dynamic combination of speech features and language context, and semantic optimization, effectively improving the recognition accuracy and semantic coherence of the speech recognition result. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention; Figure 2 This is a schematic diagram of the overall model structure of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention; Figure 3 This is a schematic diagram of the Transformer structure of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention; Figure 4 This is a schematic diagram of the self-attention mechanism structure of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention; Figure 5This is a schematic diagram of the semantic error correction module structure of the large language model in a preferred embodiment of the continuous speech recognition method based on the large language model of the present invention. Figure 6 This is a schematic diagram of the contextual conditionalization process of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention. Figure 7 This is a schematic diagram of the caching and paragraph fusion mechanism of a preferred embodiment of the continuous speech recognition method based on a large language model of the present invention; Figure 8 This is a structural diagram of a preferred embodiment of the continuous speech recognition system based on a large language model according to the present invention; Figure 9 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] With the development of smart terminals and artificial intelligence technologies, speech recognition has become an important means of human-computer interaction. Existing speech recognition systems typically rely on joint modeling of acoustic and language models to convert speech signals into text sequences. However, in scenarios involving long speech or multi-turn dialogues, these methods are prone to problems such as loss of contextual information, inaccurate segmentation, and unclear understanding of ambiguous words. Furthermore, existing speech recognition systems have limited utilization of speech features such as speech rate, emotion, and accent, making it difficult to achieve personalized recognition.

[0019] With the rise of large language models, they have shown significant advantages in context understanding and semantic reasoning. However, most current speech recognition systems only use large language models in the post-processing stage, failing to achieve deep integration with the speech recognition module. This results in poor performance in terms of real-time performance and context consistency, failing to meet user needs.

[0020] Therefore, there is an urgent need for a continuous speech recognition method that can combine speech features with language context and has dynamic semantic optimization capabilities to improve recognition accuracy and semantic coherence.

[0021] This invention addresses the problems of semantic fragmentation, missing context, and recognition delay in existing continuous speech recognition. It proposes a continuous speech recognition and semantic optimization method based on a large language model. This method can achieve automatic speech recognition, semantic error correction, and context consistency maintenance in continuous speech input or multi-turn dialogue scenarios, and can effectively improve the accuracy and naturalness of long speech recognition.

[0022] The preferred embodiment of the continuous speech recognition method based on a large language model described in this invention, such as... Figure 1 and Figure 2 As shown, the continuous speech recognition method based on a large language model includes the following steps: Step S10: Acquire the speech signal, and perform frame preprocessing and feature extraction on the speech signal to obtain short-time energy and zero-crossing rate.

[0023] This invention includes: a voice acquisition module, a dynamic detection module, a speech recognition module, a large language model optimization module, and a cache fusion module. These modules work collaboratively through data flow interaction to achieve real-time recognition and semantic optimization of continuous speech.

[0024] First, the voice acquisition module is used to acquire the input voice signal in real time and perform frame preprocessing to extract volume energy features and silence segments, thereby obtaining the audio stream and silence mask.

[0025] Specifically, the speech signal is acquired in real time and processed into frames according to a preset frame length to obtain framed speech signals; the discrete sample values ​​of the framed speech signals and the number of sampling points in each frame are obtained; the short-time energy of each frame is calculated based on the number of sampling points and the framed speech signals; and the zero-crossing rate of each frame is calculated based on the number of sampling points, the discrete sample values, and the framed speech signals. The calculation expression for the short-time energy is as follows: ; in, For the first The short-time energy of a frame, The number of sampling points per frame. Let x(n) be the number of frames, and x(n) be the framed audio signal. The formula for calculating the zero-crossing rate is: ; in, Let n be the zero-crossing rate of the nth frame. For symbolic functions, For the speech signal in the first Discrete sampled values ​​of a frame.

[0026] The specific processing procedure of the voice acquisition module is as follows: the acquired voice signal is processed in frames (e.g., 200ms per frame), the volume threshold is calculated using short-time energy and zero-crossing rate detection methods, the silence segments in the voice signal are marked, and the corresponding silence mask is generated for subsequent dynamic recognition triggering.

[0027] The formula for calculating short-time energy is as follows: ; The formula for calculating the zero-crossing rate is: .

[0028] Step S20: Perform dynamic mute detection based on the short-time energy and the zero-crossing rate to obtain the trigger audio stream.

[0029] The dynamic detection module in this invention adopts a dynamic silence detection algorithm, which performs dynamic silence detection based on short-time energy and zero-crossing rate to achieve speech segment triggering (determining which frames in the speech signal are speech frames and which are silence frames).

[0030] Specifically, an adaptive thresholding strategy is used to determine the short-time energy threshold and the zero-crossing rate threshold, as well as the shortest speech duration and the maximum silence duration.

[0031] To adapt to different noise environments, this invention employs an adaptive thresholding strategy to determine the volume threshold and zero-crossing rate threshold. The specific process is as follows: starting from the beginning of the speech, a background silence interval is selected, and its short-time energy mean is calculated. with standard deviation Based on this, the dynamic energy threshold is calculated. (i.e., the short-time energy threshold in this invention): ; Where K is an empirical coefficient (usually taken as 1.5 to 3). Similarly, the zero-crossing rate threshold... The value is determined by the average zero-crossing rate and standard deviation of the silent segment. After obtaining the short-time energy threshold and the zero-crossing rate threshold, a joint determination is performed on each frame of signal: when... > and < If a frame is silent, it is marked as a "speech frame"; otherwise, it is marked as a "silent frame". To avoid misjudgment due to transient noise or short pauses, a minimum speech duration is further introduced. With maximum silent duration Constraints. When the duration of consecutively detected speech frames exceeds... The speech segment is triggered when the duration of consecutive silence frames exceeds a certain threshold. The system can determine the end of a speech segment in a timely manner, thereby effectively suppressing false triggering caused by short noises or pauses.

[0032] Based on the shortest speech duration and the maximum silence duration, the short-time energy is compared with the short-time energy threshold to obtain a first comparison result, and the zero-crossing rate is compared with the zero-crossing rate threshold to obtain a second comparison result; based on the first comparison result and the second comparison result, speech frames and silence frames are obtained, and a binary silence mask is constructed based on the speech frames and the silence frames; The expression for the binary mute mask is as follows: ; in, To dynamically detect the status, This refers to the detection status of the previous frame. For short-time energy threshold, The zero-crossing rate threshold, The length of the currently continuously detected speech (in seconds). The shortest speech duration, The current duration of continuous silence (in seconds). This is the maximum silent duration.

[0033] Based on this, a binary silence mask M={ corresponding to the speech frame sequence is generated. , ,..., }in: ; This binary silence mask is used for subsequent dynamic recognition triggering and speech segment boundary determination. When a continuous "1" segment is detected in the mask, the system starts the end-to-end speech recognition module; when a continuous "0" segment is detected and its duration exceeds T... max When the time is up, the system will automatically end the current recognition and output the result.

[0034] Furthermore, the specific processing steps of the dynamic detection module include: comparing the short-time energy and zero-crossing rate with a preset threshold, combined with the shortest speech length T. min and maximum silent duration T max The judgment rules trigger or terminate the speech recognition process to avoid misrecognition caused by noise or short pauses.

[0035] The formula for dynamic silence detection is: .

[0036] An audio stream is constructed based on the binary silence mask, the short-time energy, the zero-crossing rate, and the framed speech signal, and the trigger audio stream is extracted from the audio stream.

[0037] It is understood that the audio stream is a sequence of speech data frames after sampling, framing, and windowing. Each frame contains speech sampling data and its characteristic parameters for a corresponding time period, used to represent the temporal variation characteristics of the speech signal, which can be simplified into a mathematical expression: ; in, ; This represents the original waveform data of the i-th frame (i.e., the framed audio signal in this invention). Represents the short-time energy of the i-th frame; This represents the zero-crossing rate of the i-th frame; This indicates a speech flag marked by a silence mask (0 or 1, where 1 represents a speech frame and 0 represents a silence frame).

[0038] Step S30: Determine the end-to-end speech recognition model, and use the end-to-end speech recognition model to transcribe the triggered audio stream into text to obtain the original text sequence. The end-to-end speech recognition model includes an encoder and a decoder.

[0039] The speech recognition module of this invention includes an end-to-end speech recognition model (i.e., an ASR acoustic model), which is used to perform preliminary transcription of the trigger audio stream to obtain the original text sequence.

[0040] Specifically, acoustic features are extracted from the triggered audio stream to obtain a frame-level acoustic feature sequence; an end-to-end speech recognition model is determined, and the frame-level acoustic feature sequence is input into the encoder in the end-to-end speech recognition model. The encoder extracts contextual speech features from the frame-level acoustic feature sequence to obtain a latent space representation; the latent space representation is input into the decoder in the end-to-end speech recognition model, and the decoder generates a symbol sequence step by step based on the latent space representation to obtain the original text sequence.

[0041] like Figure 2 As shown, the specific processing procedure of the speech recognition module in this invention is as follows: The triggered audio frames are transcribed using an end-to-end speech recognition model (such as Transformer) to obtain the original text sequence. First, the triggered audio frames determined in the dynamic detection module are processed through feature extraction to obtain a frame-level acoustic feature sequence, which is then input into the end-to-end speech recognition model. This model uses the frame-level acoustic feature sequence as input, as shown below: ; in, It is the feature vector of the t-th frame. It represents a d-dimensional real vector.

[0042] In an end-to-end speech recognition model, the encoder first extracts speech features containing contextual information to obtain the latent space representation H={h1,h2,...,h...}. T The decoder generates the output symbol sequence W={w1,w2,...,w} step by step based on an attention mechanism. N}

[0043] The meaning of stepwise generation is that the decoder generates one symbol at a time, and the generation of the next symbol depends on the previously generated symbols. The generation steps are as follows: 1. First, embed the generated symbols into vectors and add positional encoding; 2. Process the generated symbols through the decoder's self-attention to capture the internal dependencies of the sequence; 3. Combine the encoder output H with the decoder state through the cross-attention mechanism between the encoder and decoder, focusing on the audio frame information most relevant to the current generation; 4. The output is processed through a feedforward network + softmax to obtain the probability distribution of the next symbol; 5. Select the most likely symbol from the probability distribution (greedy or beam search) as the input for the next step.

[0044] The text sequence output by an end-to-end speech recognition model can be represented as: ; Where P(W|X) is the conditional probability of the output sequence W given the input speech features X. argmax represents finding the output sequence W with the highest probability. Furthermore, the conditional probability can be jointly represented by the acoustic modeling probability and the language prior probability: ; in, P(W) represents the probability that the acoustic model models the correspondence between speech and text, and P(W) represents the probability that the language model models the plausibility of the text sequence.

[0045] During training, the Connectionist Temporal Classification (CTC) loss is used, and its form is as follows: ; Here, π represents all possible alignment paths, and B denotes the path compression function. By minimizing the loss function, the model parameters are optimized, ultimately yielding a recognition model capable of real-time transcription of speech segments.

[0046] Step S40: Determine the large language model, and use the large language model to perform semantic error correction and context optimization on the original text sequence to obtain the recognized text sequence.

[0047] The large language model optimization module in this invention is used to input the original text sequence output by the speech recognition module into the large language model for semantic error correction and context optimization.

[0048] Specifically, a large language model is determined, and the original text sequence is input into the large language model; contextual semantic information is determined, and the large language model calculates the sequence maximization generation probability based on the original text sequence and the contextual semantic information to obtain the conditional probability.

[0049] like Figure 2 As shown, the large language model optimization module of this invention includes a large language model, which comprises a pre-trained model based on the Transformer architecture (such as LLaMA), to perform semantic-level optimization and context-consistency reconstruction on the original text sequence obtained by the speech recognition module. This model achieves automatic error correction, completion, and ambiguity resolution of speech-recognized text by jointly modeling language prior probabilities and context dependencies. The specific processing procedure of the large language model optimization module is as follows: First, the original text sequence obtained by the speech recognition module is processed using contextual semantic information... Under these conditions, the goal of a large language model is to maximize the conditional probability of sequence generation. The expression for the conditional probability is: ; in, For contextual semantic information, including previous recognition results or multi-turn dialogue content, P( This represents the probability of generating the t-th word given the context and preceding text. Conditional probability is used to evaluate the semantic rationality of candidate text within the context.

[0050] like Figure 3 The diagram shows the Transformer architecture, a model based on the Encoder-Decoder framework. It primarily consists of self-attention, multi-head attention, positional encoding, residual connections, and layer normalization, with self-attention being its core. This mechanism weights the dependencies between different words in the input text, enabling the model to focus on key emotional expressions in the dialogue, such as words with high emotional load like "very tired" or "don't want to talk," thereby enhancing its ability to perceive the user's current psychological state. Furthermore, self-attention maintains semantic consistency across multiple rounds of interaction, improving the system's ability to track and predict user emotional trends.

[0051] like Figure 4 The diagram shown is a structural diagram of the self-attention mechanism. The calculation steps include: 1. Input representation: Given an input sequence X containing n words, each word is represented as a d-dimensional vector, and the shape of the input matrix is ​​(n, d); 2. Calculate Query, Key, and Value: Through linear transformation, the input X is mapped to the spaces of Query (Q), Key (K), and Value (V), respectively. The calculation of Query, Key, and Value can be expressed as: ; Among them, W Q W K W VThese are the weight matrices to be learned, with shapes (d, ..., ...) d q ), (d, d k ), (d, d v ).in, d q , d k , d v represents the dimensions of the Query, Key, and Value vectors, respectively, used to define the mapping size of input features in different attention subspaces, where d represents the input dimension.

[0052] 3. Calculate attention score: The attention score for each word on the other words is obtained by calculating the dot product of the query and key. To avoid excessively large values, it is usually divided by... Scaling: Attention ; The softmax function ensures that the sum of the attention weights is 1.

[0053] 4. Weighted Summation: The attention weights are used to weight and sum the values, resulting in a new representation. The attention score matrix has a shape of (n, n), representing the attention weight of each position in the sequence to the other positions. After weighted summation, the resulting matrix has a shape of (n, n). d v Where n represents the length of the input sequence, i.e., the number of input frames or words; (n, n) is used to represent the attention weight relationship between any two positions in the sequence, (n, n) d v ) represents the feature representation of each position in the corresponding sequence after fusing the context.

[0054] To enhance the model's expressive power, Transformer uses a multi-head attention mechanism. The basic idea is to divide the self-attention mechanism into multiple "heads," each independently calculating attention, and then merging their results. The specific steps are as follows: 1. Multi-head computation: Input X is processed through different weight matrices W Q W K and W V Mapped to different Query, Key, and Value spaces, each head independently computes self-attention: ; in, , , .

[0055] 2. Concatenation Result: Concatenate the outputs of all heads together: ; in, It is a learnable linear transformation matrix used to adjust the dimensions of the concatenated matrix.

[0056] Obtain the confidence probability corresponding to the original text sequence, and calculate the comprehensive error correction score based on the confidence probability and the conditional probability; The formula for calculating the comprehensive error correction score is as follows: ; in, To calculate the overall error correction score, and All are weighted coefficients. Let be the confidence probability. The conditional probability; The text sequence with the highest score in the original text sequence is selected based on the comprehensive error correction score to obtain the recognized text sequence.

[0057] Understandably, after calculating the conditional probability of the language model, the confidence probability output by the speech recognition model (ASR) is combined with the conditional probability of the language model to form a comprehensive error correction score. The optimal sentence is then selected from multiple candidate texts based on this score, achieving semantic error correction and optimization. The formula for the comprehensive error correction score is as follows: ; in, and These are weighting coefficients used to balance the influence of acoustic and linguistic information; This is the output probability for speech recognition. For the conditional probability of the language model.

[0058] like Figure 5 The diagram illustrates the error correction and optimization process of the large language model. Specifically, this process includes inputting the original text sequence and its confidence probability obtained from the ASR speech recognition model into the large language model, while simultaneously considering the current context state. Semantic analysis and error correction are performed. Internally, the module uses a comprehensive scoring function to sort and select candidate texts. The comprehensive scoring function is as follows: ; in This represents the confidence probability of the speech recognition model. This indicates that the large language model is based on context. The probability of semantic plausibility is calculated. As a weighting coefficient, the candidate sentence with the highest score is finally selected as the optimized text (i.e., the recognized text sequence in this invention) and output to subsequent modules of the system, such as the caching and fusion module or the direct output end.

[0059] Furthermore, the context semantic information is dynamically updated to obtain updated context semantic information; The expression for the dynamic update is: ; in, For contextual semantic information, The context state of the previous round, For the text sequence identified in the previous round, Encode the emotional or speech features of the previous round (such as tone, energy, speech rate, etc.). This is a context fusion function, which can be implemented using a Transformer attention network.

[0060] Understandably, the contextual condition mechanism enables the continuous memorization of the context, ensuring the semantic continuity and emotional consistency of the multi-round recognition output, and defining the context of the current round. Dynamic update method: .

[0061] like Figure 6 The diagram illustrates the contextual conditions process, used to dynamically update the current context state during multi-round speech recognition, ensuring semantic coherence and emotional consistency in multi-round text output. For example... Figure 6 As shown, the previous context Compared with the previous round of text output and the voice or emotional characteristics of the previous round. Input to context fusion function In this process, a semantic fusion network based on the Transformer attention mechanism is used to process the data, thereby generating the current round's context state. .

[0062] Step S50: The recognized text sequence is reprocessed using a caching and paragraph fusion mechanism to obtain the final speech recognition result.

[0063] This invention includes a cache fusion module, which incorporates a cache and paragraph fusion mechanism to splice and organize multi-round recognition results, maintaining semantic continuity and paragraph integrity.

[0064] Specifically, the recognized text sequence is generated in multiple rounds based on the updated context semantic information to obtain multiple rounds of recognition results; the multiple rounds of recognition results are merged in chronological order or contextual order to obtain a continuous text stream; the continuous text stream is cleaned, standardized, semantically optimized, and contextually optimized to obtain the final speech recognition result.

[0065] Understandably, the cache fusion module, based on the output of the large language model optimization module, further enables multi-turn speech recognition or dialogue generation to automatically maintain semantic continuity and paragraph integrity in continuous dialogues or long speech segments. To this end, this invention introduces a cache and paragraph fusion mechanism to store, splice, organize, and logically optimize multi-turn recognition results to form complete and coherent text paragraphs.

[0066] In this context, multi-round recognition results refer to dividing continuous speech input into multiple triggered audio segments (or multiple rounds of user speech), performing end-to-end speech recognition separately for each round, and obtaining the recognized text sequence W for that round. i Repeating this process multiple times yields a multi-round text sequence [W1, W2, ..., W...]. n In this process, the recognition results of each round are first saved to a cache (memory or temporary storage structure). The cache can be saved in chronological or conversational order to ensure that the text order is consistent with the speech order. Then, the multi-round recognition texts in the cache are merged into a continuous text stream according to chronological or contextual order. Next, the concatenated text is cleaned and standardized (duplicate removal, correction of recognition errors, and word order adjustment, etc.). Finally, the processed text is semantically and contextually optimized to maintain semantic continuity and paragraph integrity.

[0067] like Figure 7 The diagram shows a caching and paragraph fusion mechanism. This invention first integrates the previous round's historical cache area... Compared with the current round of speech recognition results W t (Text generated by large language model optimization) Input fusion function f The caching and merging module concatenates, deduplicates, or merges text based on semantic coherence, paragraph structure, or chronological order, generating an updated cache area. The expression is: ; in, f This is a fusion function used to concatenate, deduplicat, optimize logic, and reconstruct paragraph structures from multiple rounds of text. The previous round's historical cache content does not store a fixed vector, but rather comprehensive information: ; Each of them It is the recognized text after several rounds of optimization by a large language model.

[0068] Updated cache It can be used not only for the next round of context conditionalization to generate the current context It can also serve as a complete text paragraph in the final output of the system, ensuring the continuity and semantic integrity of the multi-round speech recognition and optimization output. Compared with existing technologies, this invention discloses a continuous speech recognition and semantic optimization method and system based on a large language model, which has the following advantages: This invention combines the output of an Automatic Speech Recognition (ASR) model with the conditional probability of a large language model, and achieves semantic continuity in multi-turn speech recognition through a contextual conditionalization mechanism; further, it introduces a caching and paragraph fusion mechanism to store, splice, and organize the multi-turn recognition results, ensuring the semantic continuity and paragraph integrity of the output text in multi-turn dialogues or long speech segments; through this mechanism, automatic error correction and enhanced semantic coherence can be achieved, making the generated text more natural and accurate; the method of this invention can significantly improve recognition accuracy and text readability in multi-turn dialogues, long speech segments, and complex speech scenarios, providing a high-performance speech recognition solution for applications such as intelligent assistants, meeting recording, education, and multimedia interaction.

[0069] Furthermore, such as Figure 8 As shown, based on the above-described continuous speech recognition method based on a large language model, this invention also provides a continuous speech recognition system based on a large language model, wherein the continuous speech recognition system based on a large language model includes: The feature extraction module 51 is used to acquire the speech signal and perform frame-by-frame preprocessing and feature extraction on the speech signal to obtain the short-time energy and zero-crossing rate. Dynamic silence detection module 52 is used to perform dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a trigger audio stream; Text transcription module 53 is used to determine an end-to-end speech recognition model and perform text transcription on the trigger audio stream through the end-to-end speech recognition model to obtain the original text sequence; The text sequence optimization module 54 is used to determine a large language model and perform semantic error correction and context optimization on the original text sequence through the large language model to obtain a recognized text sequence. The speech recognition result output module 55 is used to reprocess the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result.

[0070] Furthermore, such as Figure 9As shown, based on the above-mentioned continuous speech recognition method and system based on a large language model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 9 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0071] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a continuous speech recognition program 40 based on a large language model, which can be executed by the processor 10 to implement the continuous speech recognition method based on a large language model in this application.

[0072] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the continuous speech recognition method based on a large language model.

[0073] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface.

[0074] In one embodiment, the steps of the continuous speech recognition method based on the large language model are implemented when the processor 10 executes the continuous speech recognition program 40 based on the large language model in the memory 20.

[0075] In summary, this invention provides a continuous speech recognition method, system, and terminal based on a large language model. The method includes: acquiring a speech signal, performing frame-by-frame preprocessing and feature extraction on the speech signal to obtain short-time energy and zero-crossing rate; performing dynamic silence detection based on the short-time energy and zero-crossing rate to obtain a trigger audio stream; determining an end-to-end speech recognition model and transcribing the trigger audio stream into text using the end-to-end speech recognition model to obtain an original text sequence; determining a large language model and performing semantic error correction and context optimization on the original text sequence using the large language model to obtain a recognized text sequence; and reprocessing the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result. This invention, through feature extraction and dynamic silence detection of the speech signal, can accurately extract the trigger audio stream. Furthermore, by employing deep fusion of the end-to-end speech recognition model and the large language model, it can achieve dynamic combination of speech features and language context, and semantic optimization, effectively improving the recognition accuracy and semantic coherence of the speech recognition result.

[0076] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0077] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, optical disk, etc.

[0078] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method of continuous speech recognition based on a large language model, characterized by, The continuous speech recognition method based on a large language model includes: Acquire speech signals, and perform frame-by-frame preprocessing and feature extraction on the speech signals to obtain short-time energy and zero-crossing rate; Dynamic silence detection is performed based on the short-time energy and the zero-crossing rate to obtain the trigger audio stream; An end-to-end speech recognition model is determined, and the trigger audio stream is transcribed into text using the end-to-end speech recognition model to obtain the original text sequence. A large language model is determined, and semantic error correction and context optimization are performed on the original text sequence using the large language model to obtain a recognized text sequence; The process of determining a large language model and performing semantic error correction and context optimization on the original text sequence using the large language model to obtain a recognized text sequence specifically includes: Determine a large language model and input the original text sequence into the large language model; Determine the contextual semantic information, and calculate the sequence maximization generation probability using the large language model based on the original text sequence and the contextual semantic information to obtain the conditional probability; Obtain the confidence probability corresponding to the original text sequence, and calculate the comprehensive error correction score based on the confidence probability and the conditional probability; The formula for calculating the comprehensive error correction score is as follows: ； wherein, is a comprehensive error correction score, and are weighting factors, is the confidence probability, is the conditional probability, is the contextual semantic information, is the input sequence of frame-level acoustic features, is the output sequence of symbols; Based on the comprehensive error correction score, the text sequence with the highest score in the original text sequence is selected to obtain the recognized text sequence; The recognized text sequence is reprocessed using a caching and paragraph fusion mechanism to obtain the final speech recognition result. 2.The large language model-based continuous speech recognition method of claim 1, wherein, The process of acquiring the speech signal and performing frame-by-frame preprocessing and feature extraction on the speech signal to obtain short-time energy and zero-crossing rate specifically includes: Real-time acquisition of audio signals, and segmentation of the audio signals into frames according to a preset frame length to obtain framed audio signals; The discrete sample values ​​of the framed speech signal and the number of sample points in each frame are obtained. The short-time energy of each frame is calculated based on the number of sample points and the framed speech signal. The zero-crossing rate of each frame is calculated based on the number of sample points, the discrete sample values, and the framed speech signal. The calculation expression for the short-time energy is as follows: ； in, For the first The short-time energy of a frame, The number of sampling points per frame. Let x(n) be the number of frames, and x(n) be the framed audio signal. The formula for calculating the zero-crossing rate is: ； in, Let n be the zero-crossing rate of the nth frame. For symbolic functions, For the speech signal in the first Discrete sampled values ​​of a frame.

3. The continuous speech recognition method based on a large language model according to claim 2, characterized in that, The step of performing dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain the trigger audio stream specifically includes: A dynamic silence detection algorithm is used to perform dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a binary silence mask; An audio stream is constructed based on the binary silence mask, the short-time energy, the zero-crossing rate, and the framed speech signal, and the trigger audio stream is extracted from the audio stream.

4. The continuous speech recognition method based on a large language model according to claim 1, characterized in that, The end-to-end speech recognition model includes an encoder and a decoder; The process of determining an end-to-end speech recognition model and then using that model to transcribe the triggered audio stream into text to obtain the original text sequence specifically includes: Acoustic features are extracted from the triggered audio stream to obtain a frame-level acoustic feature sequence; An end-to-end speech recognition model is determined, and the frame-level acoustic feature sequence is input into the encoder in the end-to-end speech recognition model. The encoder extracts contextual speech features from the frame-level acoustic feature sequence to obtain a latent space representation. The latent space representation is input into the decoder in the end-to-end speech recognition model. The decoder generates a symbol sequence step by step based on the latent space representation to obtain the original text sequence.

5. The continuous speech recognition method based on a large language model according to claim 1, characterized in that, The step of determining the contextual semantic information involves calculating the sequence maximization generation probability using the large language model based on the original text sequence and the contextual semantic information to obtain the conditional probability, and then further includes: The context semantic information is dynamically updated to obtain updated context semantic information; The expression for the dynamic update is: ； in, For contextual semantic information, For the contextual semantic information of the previous round, For the text sequence identified in the previous round, Encode the emotion or speech features from the previous round. This is a context fusion function.

6. The continuous speech recognition method based on a large language model according to claim 5, characterized in that, The process of reprocessing the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result specifically includes: The recognized text sequence is generated in multiple rounds based on the updated context semantic information to obtain multiple rounds of recognition results; The results of the multiple rounds of recognition are merged in chronological or contextual order to obtain a continuous text stream; The continuous text stream is cleaned, standardized, semantically optimized, and contextually optimized to obtain the final speech recognition result.

7. A continuous speech recognition system based on a large language model, characterized in that, The continuous speech recognition system based on a large language model is used to implement the continuous speech recognition method based on a large language model as described in any one of claims 1-6, wherein the continuous speech recognition system based on a large language model includes: The feature extraction module is used to acquire the speech signal and perform frame-by-frame preprocessing and feature extraction on the speech signal to obtain the short-time energy and zero-crossing rate. A dynamic silence detection module is used to perform dynamic silence detection based on the short-time energy and the zero-crossing rate to obtain a trigger audio stream; The text transcription module is used to determine the end-to-end speech recognition model and perform text transcription on the triggered audio stream through the end-to-end speech recognition model to obtain the original text sequence. The text sequence optimization module is used to determine a large language model and perform semantic error correction and context optimization on the original text sequence through the large language model to obtain a recognized text sequence. The speech recognition result output module is used to reprocess the recognized text sequence using a caching and paragraph fusion mechanism to obtain the final speech recognition result.

8. A terminal, characterized in that, The terminal includes: a memory, a processor, and a continuous speech recognition program based on a large language model stored in the memory and executable on the processor. When the continuous speech recognition program based on the large language model is executed by the processor, it implements the steps of the continuous speech recognition method based on a large language model as described in any one of claims 1-6.

Images