A real-time voice interaction method and system based on a large model

Abstract

This invention provides a real-time voice interaction method and system based on a large-scale model. It collects multi-turn historical dialogue data from users, enabling deep understanding of the context and adjusting subsequent strategies based on the evolution of historical dialogue content. Through precise construction of dynamic context and the powerful semantic understanding capabilities of the large-scale model, the system can better understand user intent and dialogue logic, generating responses that are more in line with human language habits, significantly improving the naturalness of the interaction. Optimizing the intelligent decision-making of the large-scale model based on reinforcement learning algorithms and human feedback allows the model to continuously learn during real-time interaction, adjusting response strategies according to user feedback and dialogue progress, improving the relevance, coherence, and user satisfaction of responses. By setting up an interruption mechanism, the system can effectively handle user interruption intentions during real-time voice interaction, ensuring the effectiveness of real-time voice interaction and thus improving its accuracy and fluency.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and more particularly to a real-time speech interaction method and system based on a large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, real-time voice interaction is gradually evolving from traditional manual modes towards intelligent and automated approaches. However, current technologies for real-time voice interaction still have limitations.

[0003] Firstly, simply recognizing keywords in the question's voice and triggering corresponding responses based on preset fixed scripts or rules means that if the question's content exceeds the preset range, an effective response cannot be given, leading to the interruption of the dialogue process. For different business scenarios, the existing system needs to frequently modify and update preset scripts and rules, which not only consumes a lot of manpower and resources for maintenance, but also has a long update cycle, making it difficult to quickly adapt to market changes, and frequent changes can easily introduce new errors.

[0004] Secondly, current systems can usually only make judgments based on single-turn dialogues and lack the ability to deeply understand the context of multi-turn dialogues. This makes it impossible for the system to adjust subsequent strategies according to the evolution of historical dialogue content, resulting in a break in dialogue logic and difficulty in solving complex problems or uncovering user intent.

[0005] Third, when the robot plays a reply voice message, the existing system cannot effectively handle the user's interruption intention. It either completely disallows interruptions, resulting in a poor user experience, or the interruption mechanism is too simple, which is prone to misjudgment and leads to chaotic dialogue, affecting communication efficiency and user satisfaction.

[0006] In summary, the technical problem to be solved by this invention is: how to improve the accuracy and fluency of real-time voice interaction. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a real-time voice interaction method and system based on a large model. By constructing a dynamic context input into the large model, reinforcement learning algorithms and human feedback are introduced to optimize the intelligent decision-making of the large model. This allows the large model to continuously learn during real-time interaction, adjusting its response strategy based on user feedback and dialogue progress. Furthermore, an interruption mechanism is set up to effectively handle user interruption intentions during real-time voice interaction, thereby improving the accuracy and fluency of real-time voice interaction.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] This invention provides a real-time voice interaction method based on a large model, comprising the following steps:

[0010] S1: Obtain and preprocess the user's historical dialogue data, which includes voice data, text data, and corresponding transcription and interaction data;

[0011] S2: Extract audio features from the preprocessed data, including MFCC features and prosodic features, and input them into the ASR model to convert them into text stream data. Construct a dynamic context based on the text stream data, including dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model.

[0012] S3: Based on the constructed dynamic context, a large model is loaded with pre-configured parameters for fine-tuning training, and the intelligent decision-making of the large model is optimized based on reinforcement learning algorithms and human feedback.

[0013] S4: Based on the dynamic context constructed from the user's voice data in the current scenario, combined with the large model parameters that are updated in real time to reflect the progress of the user's dialogue, the latest reply text is generated, and speech is synthesized based on the TTS model. An interruption mechanism is set up to determine the timing of playback and conduct real-time voice interaction.

[0014] In step S2, the ASR model is an end-to-end model based on the Transformer architecture. The fused feature sequence of MFCC features and prosodic features is input into the ASR model. The ASR model converts the fused feature sequence into corresponding text stream data by performing temporal modeling and acoustic modeling on the fused feature sequence.

[0015] In step S2, a dynamic context update mechanism is designed to update the content of each field in real time:

[0016] For the dialogue history field, the processed user text stream and system response are added to the dialogue history and marked with the corresponding timestamps to form a time-series record of user input and system output;

[0017] For the scene information field, the scene label is updated based on the scene switching keywords extracted from the text stream and the preset scene recognition rules. At the same time, the business knowledge corresponding to the new scene is loaded from the pre-built knowledge base. If no scene switching is detected, the current scene information remains unchanged. Real-time business knowledge related to the scene is supplemented according to the progress of the dialogue. The knowledge base includes, but is not limited to, relational databases, document databases and vector databases.

[0018] For the user profile field, new user attribute information extracted from the text stream will be added to the user profile information field. If there is a conflict with the existing user information, it will be handled according to preset rules.

[0019] For the ASR model confidence field, the confidence level corresponding to the current text stream output by the ASR model is recorded in this field, so that the system can take different strategies according to the confidence level in the future.

[0020] Furthermore, in step S2, the dialogue history length is managed based on a sliding window mechanism. When the number of dialogue rounds is too large, resulting in an excessively long history, the most recent N rounds of key dialogue content are retained, and early redundant information is deleted to ensure that the context information is within the input length limit of the large model. Invalid content in the scene information and user profile information is filtered out. After each round of dialogue, the integrity and consistency of the dynamic context are verified.

[0021] Furthermore, the execution process of S3 includes the following steps:

[0022] Construct training samples, which include dynamic context samples and human feedback data samples, wherein the feedback data samples are sorted data, rating data, and correction data;

[0023] With dynamic context as the core and the large model to be optimized as the agent, the elements of the reinforcement learning framework are defined, including state, action and environment, and a multi-dimensional reward function is designed, including relevance reward, coherence reward, task completion reward, user satisfaction reward and ASR confidence association reward.

[0024] Select a large model with preset parameters as the initial policy model, and use dynamic context-ideal response as training data to fine-tune and supervise the training of the initial policy model.

[0025] Using human feedback data samples as training samples, a reward model is trained to evaluate the quality of responses, and the output is a reward value;

[0026] Based on the initial policy model, the reward output by reinforcement learning and the reward output by the reward model are combined as reward signals. The policy model is iteratively optimized through the PPO algorithm so that the generated response is more in line with the needs of dynamic context and human preferences.

[0027] Furthermore, before iteratively optimizing the strategy model based on the PPO algorithm, a value model is first initialized to predict the expected reward value of the current dynamic context, assisting in the calculation of the advantage function. The calculation formula is as follows:

[0028] A t = G t -V(s);

[0029] In the formula, A t This represents the difference between the actual return and the expected return. A positive value indicates that the return exceeded expectations, while a negative value indicates that the return fell short of expectations. It is used to guide the direction of strategy updates. (G) tV(s) represents the cumulative reward at the current moment in a multi-turn dialogue, V(s) represents the expected reward value of the value model for the current state, and s is the dynamic context.

[0030] Preferably, the total loss function is obtained by combining the PPO algorithm and the value model, and the calculation formula is as follows:

[0031] ;

[0032] In the formula, L total Let L represent the total loss function. CLIP L represents the editing objective function used to optimize the response preferences of the strategy model. VF λ represents the value function loss, used to optimize the prediction accuracy of the value model. λ represents the value loss weight. The total loss is minimized through gradient descent to update the parameters of the new policy model and the value model.

[0033] Furthermore, an interruption mechanism is set up during real-time voice interaction, including the following steps:

[0034] While playing audio, ASR recognition is enabled in parallel, and VAD is used to detect the user's voice activity in real time to determine whether the user intends to interrupt.

[0035] If an attempt to interrupt is detected, the playback of the current TTS voice message is immediately stopped, and the following policy is executed:

[0036] a. Save the unplayed reply text and generate a simple response;

[0037] b. Based on the user's interruption, regenerate a response that integrates the unplayed content and the user's new intent;

[0038] c. Ignore unplayed content and directly respond to user interruptions;

[0039] If no interruption intent is detected from the user, ASR will be automatically activated after the current TTS voice recording has finished playing, and the system will wait for the user's next input.

[0040] In another aspect, the present invention provides a real-time voice interaction system based on a large model, comprising mutually communicating components:

[0041] Data processing module: used to acquire and preprocess user conversation data;

[0042] Feature extraction module: used to extract audio features from preprocessed data, including MFCC features and prosodic features;

[0043] The conversion module is used to convert extracted audio features into text data based on the ASR model. It is also used to synthesize the response text generated by the large model into speech based on the TTS model and play it.

[0044] Dynamic Context Construction Module: Constructs dynamic context based on text data transformed by the ASR model. The context includes dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model.

[0045] The response text generation module loads a large model for fine-tuning training based on historical dialogue data and pre-configured parameters. It optimizes the intelligent decision-making of the large model based on reinforcement learning algorithms and human feedback. It updates the parameters of the large model in real time based on the dynamic context constructed from the user's voice data in the current scene and the progress of the user's dialogue, generating the latest response text.

[0046] Furthermore, the conversion module includes an ASR unit, a TTS unit, and a voice interaction unit;

[0047] The ASR unit is used to convert the extracted audio features into text stream data based on the ASR model;

[0048] The TTS unit is used to convert the response text generated by the large model into speech based on the TTS model;

[0049] The voice interaction unit is used to play TTS-synthesized speech and has an interruption mechanism to determine the timing of playback for real-time voice interaction.

[0050] The beneficial effects of this invention are as follows:

[0051] This invention provides a real-time voice interaction method and system based on a large-scale model. It collects multi-turn historical dialogue data from users, enabling deep understanding of the context and adjusting subsequent strategies based on the evolution of historical dialogue content. Through precise construction of dynamic context and the powerful semantic understanding capabilities of the large-scale model, the system can better understand user intent and dialogue logic, generating responses that are more in line with human language habits, significantly improving the naturalness of the interaction. Optimizing the intelligent decision-making of the large-scale model based on reinforcement learning algorithms and human feedback allows the model to continuously learn in real-time interaction, adjusting response strategies according to user feedback and dialogue progress, improving the relevance, coherence, and user satisfaction of responses. By setting an interruption mechanism, the system can effectively handle user interruption intentions during real-time voice interaction, ensuring the effectiveness of real-time voice interaction. In summary, this invention improves the accuracy and fluency of real-time voice interaction and optimizes the user experience by constructing dynamic context for deep understanding of user intent, introducing reinforcement learning algorithms and human feedback to optimize the decision-making process of the large-scale model, and setting an interruption mechanism. Attached Figure Description

[0052] Figure 1 This is a flowchart of a real-time voice interaction method based on a large model according to the present invention.

[0053] Figure 2 This is a block diagram of a real-time voice interaction system based on a large model according to the present invention. Detailed Implementation

[0054] Please see Figures 1 to 2 As shown, the present invention provides a real-time voice interaction method and system based on a large model.

[0055] For details, please refer to Figure 1 As shown, this invention provides a real-time voice interaction method based on a large model, comprising the following steps:

[0056] S1: Obtain and preprocess the user's historical dialogue data, which includes voice data, text data, and corresponding transcription and interaction data;

[0057] We collect historical user dialogue data across various scenarios. Transcription data can be manually annotated or generated by ASR (Automatic Speech Recognition) analysis. We also record metadata such as dialogue scenario tags (e.g., customer service inquiries, product promotions, technical support), dialogue time, and user identifiers. We ensure that the acquired data covers users with different accents, speaking speeds, ages, and genders, as well as speech in varying noise environments, guaranteeing data diversity and representativeness.

[0058] For the acquired speech data, background noise is removed using a noise reduction algorithm to improve speech clarity. Endpoint detection algorithms, such as a dual-threshold method based on energy and zero-crossing rate, are used to identify and remove silent segments from the speech, preserving the valid speech portion. Subsequently, the speech data is standardized, normalizing the amplitude of the speech signal to the range [-1, 1] and unifying the sampling rate and quantization bits.

[0059] The transcribed text data is cleaned to remove typos, special symbols, and irrelevant redundant information. Entity recognition, such as names of people, places, products, and times, is then performed on the text data. Models such as BERT and CRF can be used, along with intent annotation to label the user's intent in each round of dialogue, such as querying, complaining, or purchasing, providing a foundation for subsequent contextual understanding.

[0060] The preprocessed speech and text data are precisely aligned by timestamp to form speech-text pairs. To address the imbalance in sample numbers across different intentions and scenarios, oversampling (increasing the number of minority class samples) or undersampling (reducing the number of majority class samples) is used for balancing. Speech data is augmented, such as by adjusting speech rate, scaling volume, and adding different types of background noise, to improve the model's robustness. Sensitive user information (such as ID card numbers, mobile phone numbers, and bank card numbers) is anonymized using methods like replacement and masking to ensure user privacy and security.

[0061] S2: Extract audio features from the preprocessed data, including MFCC features and prosodic features, and input them into the ASR model to convert them into text stream data. Construct a dynamic context based on the text stream data, including dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model.

[0062] Mel-frequency cepstral coefficients (MFCCs) are extracted from the preprocessed speech data, including static features, first-order difference features, and second-order difference features. MFCC features effectively reflect the spectral and auditory perception characteristics of speech. This invention further extracts prosodic features of the speech, including fundamental frequency (F0), speech rate, and energy, to assist in judging the user's emotion and speaking state. The MFCC features and prosodic features are concatenated along the feature dimension to form a fused feature sequence.

[0063] In addition, to improve user experience, audio features can also include acoustic features for emotion recognition, which are used to identify user emotions based on the current scene and the user's dialogue data, making the subsequent interaction data generated by the large model more human-like.

[0064] The ASR model is an end-to-end model based on the Transformer architecture. The fused feature sequence of MFCC features and prosodic features is input into the ASR model. The ASR model converts the fused feature sequence into corresponding text stream data by performing temporal and acoustic modeling on the fused feature sequence.

[0065] The Transformer encoder performs deep temporal modeling (capturing inter-frame temporal dependencies) and acoustic modeling (learning the mapping from speech features to semantics) on the fused feature sequence, outputting a contextual feature sequence containing speech and semantic information. Specifically, firstly, positional encoding is injected into the fused feature sequence to mark the temporal position of each frame's features, helping the model capture the temporal order relationship of features. Then, a multi-head self-attention mechanism layer in the encoder maps the input features to multiple subspaces. By calculating the attention weights in different subspaces, the dependencies between different frames in the feature sequence are captured, achieving temporal modeling. Finally, a feedforward neural network performs a nonlinear transformation on the features output by the self-attention mechanism, enhancing the model's nonlinear fitting ability to acoustic features, achieving acoustic modeling.

[0066] The encoder's contextual feature sequence output is used as the decoder's input. A text stream is generated character-by-character using an autoregressive approach, achieving a mapping from speech features to text. A multi-head self-attention mechanism layer captures the dependencies within the text sequence; a multi-head cross-attention mechanism layer performs attention calculations between the decoder's intermediate output and the encoder's contextual feature sequence, focusing on the parts of the speech features relevant to the current character to be generated, achieving a precise mapping from acoustic features to text. This ultimately generates a preliminary text sequence. Post-processing of the preliminary text sequence includes punctuation prediction (using specialized punctuation recovery models, such as BERT-based sequence labeling models) and syntax correction (using text correction models based on pre-trained language models) to improve the readability and accuracy of the text stream.

[0067] By leveraging the self-attention and cross-attention mechanisms of Transformer, end-to-end conversion from fused features to text stream is directly achieved, significantly improving the efficiency and accuracy of speech-to-text conversion.

[0068] A dynamic context is constructed based on the text stream data output by the ASR model. The dynamic context includes dialogue history, current dialogue scene information, user profile information, and the confidence of the ASR model, so that subsequent large models can better understand the context.

[0069] Before the conversation begins, create a structured context dictionary to store preset fields.

[0070] Conversation history field: Initialized as an empty list, used to store subsequent conversation content in chronological order.

[0071] Scene information field: The scene tag to which the current dialogue belongs, as well as scene-related business knowledge. Initial scene information can be obtained from the business configuration system.

[0072] User profile information field: Retrieves known user attribute data from the user profile system, such as name, gender, historical purchase records, preferences, membership level, etc. If it is a new user, it is initialized to empty and filled in after information is obtained in the conversation.

[0073] The confidence field of the ASR model refers to the quantitative evaluation value of the model's credibility in the correctness of the recognized text results. It is initialized to a null value and the corresponding value is recorded after the first ASR conversion.

[0074] Once the user begins speaking, the ASR model converts the speech into a text stream in real time, simultaneously outputting the confidence score of the text stream, ranging from 0 to 1. After post-processing the ASR output text stream, key information is extracted. For example, if the text stream contains new user attribute information (such as the user mentioning "I am your diamond member"), this information is extracted to update the user profile; if the text stream involves a scene change (such as the user switching from "checking bills" to "applying for installment payments"), scene keywords are extracted to update the scene information.

[0075] Among them, acoustic environment labels are introduced, that is, the background noise or channel features in the audio samples are learned and reproduced through a pre-trained deep learning model, which enables the ASR confidence judgment threshold to be dynamically adjusted in noisy environments.

[0076] Design a dynamic context update mechanism to update the content of each field in real time:

[0077] For the dialogue history field, the processed user text stream and system response are added to the dialogue history and marked with the corresponding timestamps to form a time-series record of user input and system output;

[0078] For the scene information field, the scene label is updated based on the scene switching keywords extracted from the text stream and the preset scene recognition rules. At the same time, the business knowledge corresponding to the new scene is loaded from the pre-built knowledge base. If no scene switching is detected, the current scene information remains unchanged. Real-time business knowledge related to the scene is supplemented according to the progress of the dialogue. The knowledge base includes, but is not limited to, relational databases, document databases and vector databases.

[0079] For the user profile field, new user attribute information (such as user emotion and identity recognition results) extracted from the text stream or acoustic features will be added to the user profile information field. If there is a conflict with the existing user information, it will be handled by preset rules (such as taking the latest information as the standard, or confirming it in subsequent conversations).

[0080] For the ASR model confidence field, the confidence level corresponding to the current text stream output by the ASR model is recorded in this field, so that the system can take different strategies according to the confidence level (e.g., when the confidence level is lower than 0.6, the system can actively confirm "Did you just say XX?").

[0081] The dialogue history length is managed using a sliding window mechanism. When the number of dialogue rounds is too large, resulting in an excessively long history, the most recent N rounds of key dialogue content are retained, while early redundant information is deleted, ensuring that the context information remains within the input length limit of the large model. Invalid content in scene information and user profile information, such as outdated business knowledge (old scene knowledge that is no longer relevant after scene switching) and incorrect user information (content that has been confirmed as mis-entered), is filtered to ensure the conciseness and accuracy of the context.

[0082] After each round of dialogue, the integrity and consistency of the dynamic context are verified.

[0083] Check if the sequence of "user input - system output" in the dialogue history is continuous. If there is any missing information, mark it as an anomaly and add a default explanation. Check if the scene information matches the dialogue content (e.g., the scene is "bill inquiry", but the dialogue history is full of "product consultation" content). If they do not match, correct the scene information using preset rules (e.g., redetermine the scene by combining high-frequency keywords in the text stream). Evaluate the reliability of user input based on the confidence level of the ASR model. If the confidence level is low and may affect contextual understanding (e.g., fuzzy recognition of key information), mark "information to be confirmed" in the context status and prompt the system to clarify it in subsequent dialogues.

[0084] S3: Based on the constructed dynamic context, a large model is loaded with pre-configured parameters for fine-tuning training, and the intelligent decision-making of the large model is optimized based on reinforcement learning algorithms and human feedback.

[0085] The execution process of S3 includes the following steps:

[0086] Construct training samples, which include dynamic context samples and human feedback data samples, wherein the feedback data samples are sorted data, rating data, and correction data;

[0087] The user input-ideal response associated with the dynamic context constructed in step S2 is used as training samples. Each sample needs to contain the complete dynamic context and the human-annotated ideal response text. Human feedback data is collected for the responses of the large model, including ranking data (3-5 different responses generated by the model under the same dynamic context are ranked according to their degree of conformity to human preferences); scoring data (each response is scored from 1 to 5 points based on four dimensions: relevance, coherence, task completion, and naturalness); and correction data (annotators modify unsatisfactory responses to form "original response - corrected response").

[0088] With dynamic context as the core and the large model to be optimized as the agent, the elements of the reinforcement learning framework are defined, including state, action and environment, and a multi-dimensional reward function is designed, including relevance reward, coherence reward, task completion reward, user satisfaction reward and ASR confidence association reward.

[0089] State definition: Current dynamic context information;

[0090] Action definition: Generate response text based on the current context;

[0091] Environment definition: Simulate a user interaction environment, receive responses generated by the intelligent agent, and output user feedback (which can be based on historical dialogue data to build a simulated user model or collected from real user feedback in actual interactions).

[0092] The multi-dimensional reward function specifically includes:

[0093] Relevance reward: The semantic similarity between the response text and the dynamic context (especially the current user input and scene information) is calculated using the Sentence-BERT model. The higher the similarity, the higher the reward value, with a range of [0,1].

[0094] Coherence reward: Evaluate the logical coherence between the generated response text and the dialogue history. Concatenate the complete dialogue history and the model's response into a complete text sequence, and input it into a pre-trained language model (such as GPT-2) to calculate the perplexity of the sequence. The lower the perplexity, the higher the reward.

[0095] Task completion reward: For scenarios with clear goals, the contribution of responses to task completion is quantified. The completion path of the scenario task is preset, and each stage is assigned a corresponding weight. The current stage of the task is determined based on the dynamic context. If the response promotes the task to the next stage, a stage reward is given. If the task is completed, the highest reward is given.

[0096] User satisfaction reward: The actual acceptance of the response is measured through explicit or implicit user feedback. For explicit feedback, if the environment returns a clear evaluation from the user, it is directly mapped to a reward value, such as 1 for satisfaction, 0.5 for general satisfaction, and 0 for dissatisfaction. For implicit feedback, it is judged based on the dialogue behavior. For example, if the user continues to ask questions to indicate acceptance of the response, a reward of 0.7 is given; if the user hangs up directly to indicate dissatisfaction, a reward of 0 is given; and if the dialogue duration is within a reasonable range, a reward of 0.5 is given.

[0097] ASR confidence-related reward: When the ASR recognition result is unreliable, the model is encouraged to actively clarify and avoid responding based on erroneous information. When the confidence level is less than or equal to 0.6 (preset threshold), if the response includes active confirmation, the reward value is 0.5; if no confirmation is made and the response is given directly, the reward value is 0. When the confidence level is greater than 0.6, a fixed reward value is set.

[0098] Select a large model with preset parameters as the initial policy model, and use dynamic context-ideal response as training data to fine-tune and supervise the training of the initial policy model.

[0099] Choose a large model suitable for real-time voice interaction scenarios and call it via API interfaces, such as the GPT series or LLaMA series. It should be noted that this large model can be a public large model service deployed in the cloud, called through its provided API interface; or it can be a large model privately deployed by the enterprise in a local or private cloud environment according to its own needs, called through an internal interface, thus ensuring data security and system independence. Pre-set some relevant parameters of the large model, such as `temperature` to control the randomness and creativity of the model output, `top_p` (kernel sampling) to control the range of candidate words in the output, `max_length` (maximum generated length) to limit the length of the system's response text, and `stop_token` (stop token) to set the marker word for the end of the dialogue; when this word appears in the generated text, generation stops, ensuring the response is concise and clear. These parameters affect the style and quality of the generated text. Using dynamic context and ideal response as training data, the model is optimized through supervised learning. The optimization objective is to minimize the cross-entropy loss between the model-generated response and the ideal response text. Mini-batch gradient descent is used, combined with an early stopping strategy to avoid overfitting. Finally, a fine-tuned initial strategy model is obtained, at which point the model can generate responses that conform to basic logic based on dynamic context.

[0100] Using human feedback data samples as training samples, a reward model for evaluating response quality is trained, and the output is the human preferred reward value;

[0101] The dynamic context text and the response text generated by the policy model are concatenated into a sequence. A pre-trained language model (such as BERT) is used as the backbone, and the output layer is mapped to a single-value reward through a linear transformation. The model is trained using contrastive loss (such as ranking loss) to learn to distinguish between good and bad responses. Finally, a reward model is obtained that can output human preference reward values ​​based on dynamic context and response text. The output reward value is specifically used to measure the degree of alignment between the response and human preferences.

[0102] Based on the initial policy model, the reward output by reinforcement learning and the reward output by the reward model are combined as reward signals. The policy model is iteratively optimized through the PPO algorithm so that the generated response is more in line with the needs of dynamic context and human preferences.

[0103] Specifically, the finely tuned initial strategy model, after loading the basic generation parameters, is used as the old strategy model to be optimized, i.e., the initial strategy model. PPO hyperparameters are set, such as 20 iterations, 32 batch size, 0.2 clipping coefficient ε, 3e-5 learning rate, and reward weights (e.g., human preference reward weight α=0.6, multi-dimensional reward weight β=0.4, which can be adjusted according to the specific scenario).

[0104] Prepare a dynamic context sample pool and randomly select samples from it as the initial input for each round of interaction. The old strategy model generates response text based on the dynamic context, that is, it generates 1-3 candidate responses through autoregressive decoding, selects the response with the highest probability as the current action, and records the probability distribution when the response is generated (i.e. the output probability of the model for each token) for subsequent calculation of the probability ratio of the new and old strategies.

[0105] The generated response is input into the environment to obtain environmental feedback. Based on the dynamic context and the response, the reward values ​​for each item are calculated and summed to obtain the multi-dimensional reward value R. multi (Normalized to the 0-1 range). The dynamic context and generated response are input into the reward model to obtain the human-preferred reward R. human Calculate the total reward R total The final reward signal for the current response is calculated using the following formula:

[0106] .

[0107] Each interaction trajectory is stored in the format of (dynamic context, generated response, action probability distribution, total reward, environmental feedback), and accumulated in the current round of sampling data buffer until the preset sample size is reached.

[0108] Apply a time discount to the total reward for each trajectory (if it's a multi-turn dialogue, consider the present value of future rewards); for single-turn dialogues, use R directly. totalAs an immediate reward, multi-turn dialogues calculate cumulative rewards, using the following formula:

[0109] G t =R total +γ×G t+1 ;

[0110] In the formula, G t G represents the cumulative reward at the current moment across multiple rounds of dialogue. t+1 For the cumulative reward in the next moment of a multi-round dialogue, R total The initial reward is γ, which is a discount factor, such as 0.95, to make the model focus more on long-term rewards rather than short-term feedback.

[0111] Before iteratively optimizing the strategy model based on the PPO algorithm, a value model is first initialized to predict the expected reward value of the current dynamic context and to assist in calculating the advantage function. The calculation formula is as follows:

[0112] A t = G t -V(s);

[0113] In the formula, A t This represents the difference between the actual return and the expected return. A positive value indicates that the return exceeded expectations, while a negative value indicates that the return fell short of expectations. It is used to guide the direction of strategy updates. (G) t V(s) represents the cumulative reward at the current moment in a multi-turn dialogue, V(s) represents the expected reward value of the value model for the current state, and s is the dynamic context.

[0114] Based on the PPO policy update, the trajectory data in the buffer is first converted into training samples: (dynamic context s, action a, i.e., response text, old policy action probability π) old (a|s), advantage A t , Reward G t The parameters are then shuffled and divided into multiple batches. Forward computation is performed on the new strategy model, i.e., the parameters of the old strategy model are loaded as the initial parameters of the new strategy model. The dynamic context s and action a are input, and the probability π of the new strategy action is calculated. new (a|s). Calculate the probability ratio, used to measure the change in preference for the same response under the new strategy. The formula is:

[0115] r t =π new (a|s) / π old (a|s);

[0116] In the formula, r t Let r be the probability ratio. t >1 indicates that the new strategy favors the current action; if r t<1 indicates that the new strategy dislikes the current action more, π old (a|s) represents the probability of the action under the old policy, π new (a|s) represents the probability of the new strategy action.

[0117] Construct the core objective function of PPO, maximize this function to update the new strategy, and calculate the formula as follows:

[0118] ;

[0119] In the formula, L CLIP This represents the clipping objective function, used to optimize the response preference of the strategy model, clip(r) t (1-ε, 1+ε) is the cutoff function, which means that r is cut off. t Restricted to the interval [1-ε, 1+ε], E represents the expectation over all training samples.

[0120] If A t >0 indicates a response exceeding expectations, limiting r t ≤1+ε, to avoid the new strategy from over-amplifying the probability of this action and to prevent the update magnitude from being too large;

[0121] If A t <0 indicates the response did not exceed expectations, limiting r. t ≥1-ε, to avoid the new strategy from excessively reducing the probability of the action and to ensure the stability of the strategy;

[0122] Simultaneously optimize the value model by updating its parameters using MSE loss, so that the predicted value V... new (s) Closer to actual return G t The calculation formula is:

[0123] ;

[0124] In the formula, L VF V represents the value function loss, used to optimize the prediction accuracy of the value model. new (s) represents the predicted reward of the value model for the current dynamic context s, used to calculate the advantage function A. t .

[0125] The combined PPO algorithm and value model yield the total loss function, calculated as follows:

[0126] ;

[0127] In the formula, L total Let L represent the total loss function. CLIP L represents the editing objective function used to optimize the response preferences of the strategy model. VFλ represents the value function loss, used to optimize the prediction accuracy of the value model. λ represents the value loss weight. For example, if the value is 0.5, the total loss is minimized through gradient descent, and the parameters of the new policy model and value model are updated.

[0128] Each batch of data is trained for 1-3 epochs (to avoid overfitting), and then the next batch is started, until all data in the current round is trained. After one iteration is completed, the parameters of the new policy model are copied to the old policy model to provide a benchmark for the next iteration.

[0129] S4: Based on the dynamic context constructed from the user's voice data in the current scenario, combined with the large model parameters that are updated in real time to reflect the progress of the user's dialogue, the latest reply text is generated, and speech is synthesized based on the TTS model. An interruption mechanism is set up to determine the timing of playback and conduct real-time voice interaction.

[0130] The updated large model parameters include generation parameters, inference parameters, and model weights, such as temperature, top_p, and max_length.

[0131] During real-time voice interaction, when a user speaks, the user's speech is converted into a text stream through ASR and the text is immediately added to the dialogue history of the dynamic context. At the same time, the context state is updated. Combining the current dialogue scenario and the user's real-time feedback, the scenario information and user information are dynamically updated to ensure that the context can accurately reflect the current dialogue state.

[0132] The generation parameters of the large model are adjusted in real time based on the current progress of the user dialogue, for example:

[0133] When a user's emotion is detected to be urgent, the temperature value can be lowered to make the output response more certain and concise. Conversely, when the user's emotion is calm, the temperature value can be appropriately increased to enhance the diversity of responses. Specifically, a voice emotion recognition model analyzes the user's speech rate, tone, energy, and other characteristics to determine the user's emotion.

[0134] When the conversation enters the stage of confirming key information (such as confirming user identity information), the top_p value should be increased to ensure that the key information output is accurate; when the conversation is in the stage of casual chat or recommendation, the top_p value can be decreased to make the response more targeted.

[0135] Adjust max_length according to the user's speaking speed. For users who speak quickly, shorten max_length appropriately to avoid excessively long replies; for users who speak slowly, increase max_length appropriately to provide more detailed information.

[0136] Parameter update rules can be implemented through preset strategies (based on business experience) or machine learning models (training parameters and adjusting the model using historical data) to ensure that parameter adjustments meet current interaction requirements.

[0137] The dynamically constructed context is input into a trained and optimized large model. This model, combined with real-time updated generation parameters, generates the latest response text. Specifically, the large model generates multiple candidate responses based on the updated parameters, selects the optimal response from the candidates according to preset rules, and outputs the final response. This response can be text for speech synthesis or a text message directly for display. Speech is synthesized based on TTS, or text is directly output for real-time interaction. The response text generated by the large model undergoes content review to filter sensitive information and ensure compliance. If sensitive information is found, a preset security response is triggered (e.g., "Sorry, this content cannot be provided to you"). The approved response text is then input into the TTS model. The TTS model performs prosodic prediction (predicting pitch, speech rate, pauses, etc.) and acoustic synthesis based on the text content to generate a natural and fluent speech signal. The TTS model can be an end-to-end neural TTS model (such as VITS, Tacotron2+WaveNet), supports custom speakers (such as setting different genders, ages, and styles of voice), and can adjust the voice style according to the dialogue scenario (such as using a friendly and gentle voice in customer service scenarios and a clear and formal voice in notification scenarios).

[0138] The interruption mechanism in real-time voice interaction includes the following steps:

[0139] While playing audio, ASR recognition is enabled in parallel, and VAD is used to detect the user's voice activity in real time to determine whether the user intends to interrupt.

[0140] The system uses VAD (Voice Activity Detection) technology to detect users' voice activity in real time. When the detected user's voice energy exceeds a preset threshold and the duration exceeds a set duration (e.g., 300ms), it is initially determined that the user may interrupt. Combined with the real-time recognition results of the user's interrupted speech by ASR (Automatic Speech Detection), if valid text content (not noise or unintentional speech) is identified, it is confirmed that the user has a clear intention to interrupt.

[0141] If an attempt to interrupt is detected, the playback of the current TTS voice message is immediately stopped, and the following policy is executed:

[0142] a. Save the unplayed reply text and generate a simple response;

[0143] b. Based on the user's interruption, regenerate a response that integrates the unplayed content and the user's new intent;

[0144] c. Ignore unplayed content and directly respond to user interruptions;

[0145] If no interruption intent is detected from the user, ASR will be automatically activated after the current TTS voice recording has finished playing, and the system will wait for the user's next input.

[0146] For special scenarios, such as playing important notifications or security alerts, an uninterrupted mode can be set to ensure that key information is conveyed completely. In this case, even if user voice activity is detected, the voice will continue to play, and the user input will be responded to after the playback is complete.

[0147] On the other hand, such as Figure 2 As shown, this invention provides a real-time voice interaction system based on a large model, comprising mutually communicating components:

[0148] Data processing module: used to acquire and preprocess user conversation data;

[0149] Feature extraction module: used to extract audio features from preprocessed data, including MFCC features and prosodic features;

[0150] The conversion module is used to convert extracted audio features into text data based on the ASR model. It is also used to synthesize the response text generated by the large model into speech based on the TTS model and play it.

[0151] Dynamic Context Construction Module: Constructs dynamic context based on text data transformed by the ASR model. The context includes dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model.

[0152] The response text generation module loads a large model for fine-tuning training based on historical dialogue data and pre-configured parameters. It optimizes the intelligent decision-making of the large model based on reinforcement learning algorithms and human feedback. It updates the parameters of the large model in real time based on the dynamic context constructed from the user's voice data in the current scene and the progress of the user's dialogue, generating the latest response text.

[0153] The conversion module includes an ASR unit, a TTS unit, and a voice interaction unit.

[0154] The ASR unit is used to convert the extracted audio features into text stream data based on the ASR model;

[0155] The TTS unit is used to convert the response text generated by the large model into speech based on the TTS model;

[0156] The voice interaction unit is used to play TTS-synthesized speech and has an interruption mechanism to determine the timing of playback for real-time voice interaction.

[0157] The following is a detailed and specific embodiment of how the real-time voice interaction system of the present invention is applied in an intelligent outbound calling scenario:

[0158] Based on the specific scenarios of outbound calls, the corresponding persona, voice, script, and variables are configured for the large model. A call control module is added for the system to interact with the Freeswitch platform and control various commands.

[0159] First, the system sends a call command to Freeswitch via the call control module. Once the user answers the call, the call officially begins. The system plays an opening statement according to the configured script. If an audio file exists, the system sends a precise command to Freeswitch to play it; otherwise, the system sends the opening statement text, pre-filled with script variables, along with the specified speaker information, to Freeswitch, which then uses TTS (Text-to-Speech) to synthesize and play the speech in real time. The opening statement playback is set to be uninterrupted to ensure that key opening information is conveyed completely and clearly to the user. Only after receiving a notification from Freeswitch that the opening statement playback is complete does the system store the robot's opening statement in its memory cache as the first dialogue round record of the call.

[0160] After the opening remarks are played, the "Listen-Think-Speak" segment begins to facilitate a natural and fluent dialogue with the user.

[0161] The first step, "listening": intelligently recognizing speech and capturing user intent in real time.

[0162] The system sends a command to Freeswitch to enable ASR monitoring and accurately recognize the user's voice input, converting it into a text stream in real time. When ASR recognizes that the user has finished speaking, Freeswitch will trigger a speech recognition completion event. The system receives this event, obtains the complete text of the user's speech, and caches it.

[0163] During this process, a speech recognition timeout is set. If the ASR does not recognize any valid speech input within a preset time, or remains silent, Freeswitch will return an empty string. Based on this, the system determines that the customer is in a silent state and accurately accumulates the number of consecutive silences. If a preset threshold is reached, the system will trigger a forced hang-up operation to save resources.

[0164] While ASR is enabled, VAD is started in parallel. If highly active speech is detected in a short period of time, the system will perform preliminary phrase recognition. This optimization is used to judge the customer's interruption intention in advance and shorten the response time.

[0165] The second step, "Thinking": Dynamic context construction and response text generation.

[0166] Select a suitable large model for the current outbound calling scenario and call it through the API interface. After fine-tuning and training based on preset parameters and user history dialogue data, optimize the intelligent decision-making of the large model based on reinforcement learning algorithms and human feedback. The system constructs a dynamic context based on the recognized user text, inputs it into the optimized large model for inference, and generates the latest response text based on the progress of the dialogue and the real-time updated model parameters. The system receives the text returned by the large model, performs secondary verification and formatting to ensure that it meets the requirements of the speech synthesis engine and improves the stability of speech playback.

[0167] The third step, "speaking": intelligent speech synthesis with an interruption mechanism.

[0168] After receiving the reply text, the system immediately sends the reply text and the speaker information specified in the script to Freeswitch, which then calls the TTS service to synthesize the text into natural and fluent speech for playback in real time.

[0169] The robot is designed with an interruption mechanism during voice playback (except for the opening and closing remarks). The system simultaneously uses ASR (Automatic Speech Recognition) and VAD (Voice-Aware Audio) to monitor for user interruption intentions. When the system receives a speech recognition start event from Freeswitch, it immediately starts a high-precision timer. If, within the timer's set period, the system does not receive a speech recognition completion event from Freeswitch, it determines that the user is still speaking and has a clear intention to interrupt. At this point, the system immediately sends a command to Freeswitch to interrupt the current playback.

[0170] The system will then receive an event indicating that the speech synthesis playback, which was interrupted prematurely, has finished. To provide a better user experience, the system will immediately play a preset, uninterrupted prompt (e.g., "Sorry, I didn't hear you clearly, could you please repeat it?"). Multiple prompts can be configured and can be played in a round-robin or random manner to avoid repetition and increase the naturalness and friendliness of the conversation. After the prompt finishes playing, the system will restart speech recognition, precisely waiting for the user to rephrase their statement, and enter a new "listening" phase.

[0171] If the system receives a speech recognition completion event from Freeswitch within the timer's set time (usually meaning the customer only made a brief sound or cleared their throat, without intending to interrupt), the system confirms that the customer did not intend to interrupt. The system will discard this brief speech recognition result and continue playing the robot's current speech. After the current speech finishes playing, the system will restart speech recognition and enter the normal "listening" phase to ensure the smoothness of the conversation.

[0172] Once the robot's voice message finishes playing (or finishes playing due to an interruption), the system will return to "listen" mode, restart voice recognition, and wait for the customer to speak. The entire "listen-think-speak" cycle will continue until the call is terminated by a preset hang-up strategy (such as reaching the maximum call duration, consecutive silence counts, automatic hang-up rounds, or the large model output containing hang-up trigger words).

[0173] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A real-time voice interaction method based on a large model, characterized in that, Includes the following steps: S1: Obtain and preprocess the user's historical dialogue data, which includes voice data, text data, and corresponding transcription and interaction data; S2: Extract audio features from the preprocessed data, including MFCC features and prosodic features, and input them into the ASR model to convert them into text stream data. Construct a dynamic context based on the text stream data, including dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model. S3: Based on the constructed dynamic context, a large model is loaded with pre-configured parameters for fine-tuning training, and the intelligent decision-making of the large model is optimized based on reinforcement learning algorithms and human feedback. S4: Based on the dynamic context constructed from the user's voice data in the current scenario, combined with the large model parameters that are updated in real time to reflect the progress of the user's dialogue, the latest reply text is generated, and the speech is synthesized based on the TTS model. An interruption mechanism is set up to determine the timing of playback and to conduct real-time voice interaction. In step S2, the ASR model is an end-to-end model based on the Transformer architecture. The fused feature sequence of MFCC features and prosodic features is input into the ASR model. The ASR model converts the fused feature sequence into corresponding text stream data by performing temporal modeling and acoustic modeling on the fused feature sequence. In step S2, a dynamic context update mechanism is designed to update the content of each field in real time: For the dialogue history field, the processed user text stream and system response are added to the dialogue history and marked with the corresponding timestamps to form a time-series record of user input and system output; For the scene information field, the scene label is updated based on the scene switching keywords extracted from the text stream and the preset scene recognition rules. At the same time, the business knowledge corresponding to the new scene is loaded from the pre-built knowledge base. If no scene switching is detected, the current scene information remains unchanged. Real-time business knowledge related to the scene is supplemented according to the progress of the dialogue. The knowledge base includes, but is not limited to, relational databases, document databases and vector databases. For the user profile field, new user attribute information extracted from the text stream will be added to the user profile information field. If there is a conflict with the existing user information, it will be handled according to preset rules. For the ASR model confidence field, the confidence level corresponding to the current text stream output by the ASR model is recorded in this field, so that the system can take different strategies according to the confidence level in the future. In step S2, the dialogue history length is managed based on the sliding window mechanism. When the number of dialogue rounds is too large and the history is too long, the key dialogue content of the most recent N rounds is retained, and early redundant information is deleted to ensure that the context information is within the input length limit of the large model. Invalid content in scene information and user profile information is filtered out. After each round of dialogue, the integrity and consistency of the dynamic context are verified. The execution process of S3 includes the following steps: Construct training samples, which include dynamic context samples and human feedback data samples, wherein the feedback data samples are sorted data, rating data, and correction data; With dynamic context as the core and the large model to be optimized as the agent, the elements of the reinforcement learning framework are defined, including state, action and environment, and a multi-dimensional reward function is designed, including relevance reward, coherence reward, task completion reward, user satisfaction reward and ASR confidence association reward. Select a large model with preset parameters as the initial policy model, and use dynamic context-ideal response as training data to fine-tune and supervise the training of the initial policy model. Using human feedback data samples as training samples, a reward model is trained to evaluate the quality of responses, and the output is a reward value; Based on the initial policy model, the reward output by reinforcement learning and the reward output by the reward model are combined as reward signals. The policy model is iteratively optimized through the PPO algorithm so that the generated response is more in line with the dynamic context and human preferences. Setting up an interruption mechanism during real-time voice interaction includes the following steps: While playing audio, ASR recognition is enabled in parallel, and VAD is used to detect the user's voice activity in real time to determine whether the user intends to interrupt. If an attempt to interrupt is detected, the playback of the current TTS voice message is immediately stopped, and the following policy is executed: a. Save the unplayed reply text and generate a simple response; b. Based on the user's interruption, regenerate a response that integrates the unplayed content and the user's new intent; c. Ignore unplayed content and directly respond to user interruptions; If no interruption intent is detected from the user, ASR will be automatically activated after the current TTS voice recording has finished playing, and the system will wait for the user's next input.

2. The real-time voice interaction method based on a large model according to claim 1, characterized in that, Before iteratively optimizing the strategy model based on the PPO algorithm, a value model is first initialized to predict the expected reward value of the current dynamic context and to assist in calculating the advantage function. The calculation formula is as follows: A t = G t -V(s); In the formula, A t This represents the difference between the actual return and the expected return. A positive value indicates that the return exceeded expectations, while a negative value indicates that the return fell short of expectations. It is used to guide the direction of strategy updates. (G) t V(s) represents the cumulative reward at the current moment in a multi-turn dialogue, V(s) represents the expected reward value of the value model for the current state, and s is the dynamic context.

3. The real-time voice interaction method based on a large model according to claim 2, characterized in that, The combined PPO algorithm and value model yield the total loss function, calculated as follows: ； In the formula, L total Let L represent the total loss function. CLIP L represents the editing objective function used to optimize the response preferences of the strategy model. VF λ represents the value function loss, used to optimize the prediction accuracy of the value model. λ represents the value loss weight. The total loss is minimized through gradient descent to update the parameters of the new policy model and the value model.

4. A real-time voice interaction system based on a large model, employing the real-time voice interaction method based on a large model as described in any one of claims 1-3, characterized in that, Including mutual communication: Data processing module: used to acquire and preprocess user conversation data; Feature extraction module: used to extract audio features from preprocessed data, including MFCC features and prosodic features; The conversion module is used to convert extracted audio features into text data based on the ASR model. It is also used to synthesize the response text generated by the large model into speech based on the TTS model and play it. Dynamic Context Construction Module: Constructs dynamic context based on text data transformed by the ASR model. The context includes dialogue history, current dialogue scene information, user profile information, and the confidence level of the ASR model. The response text generation module loads a large model for fine-tuning training based on historical dialogue data and pre-configured parameters. It optimizes the intelligent decision-making of the large model based on reinforcement learning algorithms and human feedback. It updates the parameters of the large model in real time based on the dynamic context constructed from the user's voice data in the current scene and the progress of the user's dialogue, generating the latest response text.

5. A real-time voice interaction system based on a large model according to claim 4, characterized in that, The conversion module includes an ASR unit, a TTS unit, and a voice interaction unit; The ASR unit is used to convert the extracted audio features into text stream data based on the ASR model; The TTS unit is used to convert the response text generated by the large model into speech based on the TTS model; The voice interaction unit is used to play TTS-synthesized speech and has an interruption mechanism to determine the timing of playback for real-time voice interaction.

Images