Method and system for emotion smoothing evolution speech synthesis based on large model driving

Abstract

The application discloses a kind of based on big model drive emotional smooth evolution speech synthesis method and system, the method includes: obtaining real-time voice data and corresponding historical time period of conversation text sent by outbound user;Based on speech recognition model, the speech emotion feature corresponding to the real-time voice data is identified;Based on big language model, according to the speech emotion feature and the conversation text, the next adaptive dialogue and dialogue emotion are predicted;According to the next adaptive dialogue and dialogue emotion, based on speech generation algorithm, the next outbound voice corresponding to the outbound user is generated;The next outbound voice is used to send to the outbound user to continue dialogue.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for emotion-smooth evolution speech synthesis based on a large model. Background Technology

[0002] With the rapid adoption of intelligent outbound calling systems in large-scale marketing, customer service follow-up, and targeted outreach, outbound calling platform operators are increasingly focusing on improving the coherence and user adaptability of conversations through emotional intelligence technology. Among these challenges, reducing the risk of communication interruptions has become a key technical issue. Existing technologies typically acquire real-time voice data from outbound users, generate response content using fixed script templates or simple emotion classification models, and play it back based on pre-recorded voice or basic speech synthesis technology to support call continuation. However, existing solutions lack in-depth correlation analysis of historical dialogue text, large language model-driven next dialogue and emotion prediction, and a voice generation mechanism for smooth emotional evolution. This makes it difficult to achieve natural transitions and personalized adaptation of dialogue emotions, resulting in insufficient coherence and user adaptability in outbound conversations. Mismatches in emotions or intentions can easily lead to user resistance, communication interruptions, or call failures, limiting the overall communication efficiency and success rate of intelligent outbound calling systems. Therefore, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for emotion-smooth evolution speech synthesis based on a large model, which can realize emotion-smooth evolution speech synthesis based on a large model to support intelligent outbound call continuation, improve the coherence and user adaptability of outbound dialogue, and reduce the risk of communication interruption caused by mismatch of emotions or intentions.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a large-model-driven emotion-smoothing speech synthesis method, the method comprising: Obtain real-time voice data sent by outbound users and corresponding historical dialogue text for the same time period; Based on the speech recognition model, the speech emotion features corresponding to the real-time speech data are identified. Based on the large language model, the next adapted dialogue and dialogue emotion are predicted according to the voice emotion features and the dialogue text. Based on the next adapted dialogue and dialogue sentiment, a next outbound call voice is generated for the outbound call user using a voice generation algorithm; the next outbound call voice is sent to the outbound call user to continue the dialogue.

[0005] As an optional implementation, in the first aspect of the present invention, the step of identifying the voice emotion features corresponding to the real-time voice data based on a speech recognition model includes: Based on a speech segmentation algorithm, the real-time speech data is segmented into multiple spoken voices; Each spoken voice is input into a trained emotion recognition model to obtain the output voice emotion; Based on the voice emotions corresponding to all the spoken voices, the voice emotion features corresponding to the real-time voice data are determined.

[0006] As an optional implementation, in the first aspect of the present invention, the step of segmenting multiple spoken voices corresponding to the real-time voice data based on a speech segmentation algorithm includes: Based on the speech frequency analysis algorithm, the speech frequency change information corresponding to the real-time speech data is determined; Based on the peak frequency recognition algorithm, multiple frequency peaks in the speech frequency change information are determined; Cluster all frequency points that are greater than the corresponding frequency threshold before and after each frequency peak point to obtain multiple frequency point sets; Each frequency point set corresponds to a set of speech data corresponding to all time points, which is then used to define a single speech voice.

[0007] As an optional implementation, in the first aspect of the present invention, determining the voice emotion features corresponding to the real-time voice data based on the voice emotions corresponding to all the spoken voices includes: Generate a graph data node corresponding to each spoken voice, and determine the voice emotion corresponding to the spoken voice as the node feature corresponding to the graph data node; Calculate the time difference between any two of the spoken voices at any given time point; Calculate the speech similarity between any two of the spoken voices; The weighted sum of the time difference and the speech similarity is determined as the node association feature between the graph data nodes corresponding to the two spoken voices; All the graph data nodes and their corresponding node features and node association features are determined as the voice emotion features corresponding to the real-time voice data.

[0008] As an optional implementation, in the first aspect of the invention, the step of predicting the next adapted dialogue and dialogue emotion based on the large language model, according to the speech emotion features and the dialogue text, includes: The dialogue text is input into the trained text direction prediction model to obtain the predicted text direction; The voice emotion features are input into a trained emotion description prediction graph neural network to obtain the output predicted emotion description. Based on the large language model, the next adapted dialogue and dialogue sentiment are predicted according to the predicted text direction and the predicted sentiment description.

[0009] As an optional implementation, in the first aspect of the invention, the step of predicting the next adapted dialogue and dialogue sentiment based on a large language model, according to the predicted text direction and the predicted sentiment description, includes: Based on the preset prompt word engineering, corresponding prompt word text is assembled according to the predicted text direction and the predicted sentiment description; the prompt words are used to describe the strategy requirements for responding to users in the predicted sentiment description within the predicted text direction; The prompt text is input into the trained LLM model to obtain the next adapted dialogue and dialogue sentiment.

[0010] As an optional implementation, in the first aspect of the present invention, generating the next outbound call voice corresponding to the outbound call user based on the next adapted dialogue and dialogue emotion using a speech generation algorithm includes: Determine the emotion tag corresponding to the dialogue emotion; The next adapted dialogue and the emotion tag are input into the trained speech generation algorithm to generate the next outbound call speech corresponding to the outbound call user.

[0011] As an optional implementation, in the first aspect of the present invention, determining the emotion tag corresponding to the dialogue emotion includes: Determine whether the emotion in the dialogue has a matching emotion tag in the emotion tag library. If it does, directly identify it as the corresponding emotion tag. If it does not exist, calculate the emotion similarity between the template speech corresponding to each candidate emotion tag in the emotion tag library and the real-time speech data. The candidate emotion tag with the lowest emotion similarity is determined as the emotion tag corresponding to the dialogue emotion.

[0012] A second aspect of this invention discloses a large-model-driven emotion-smoothing speech synthesis system, the system comprising: The acquisition module is used to acquire real-time voice data sent by outbound call users and the corresponding historical dialogue text for the same time period. The recognition module is used to identify the voice emotion features corresponding to the real-time voice data based on the voice recognition model; The prediction module is used to predict the next adapted dialogue and dialogue emotion based on the speech emotion features and the dialogue text, using a large language model. The generation module is used to generate the next outbound call voice corresponding to the outbound call user based on the next adapted dialogue and dialogue emotion, using a voice generation algorithm; the next outbound call voice is used to send to the outbound call user to continue the dialogue.

[0013] As an optional implementation, in a second aspect of the present invention, the specific method by which the recognition module identifies the voice emotion features corresponding to the real-time voice data based on a speech recognition model includes: Based on a speech segmentation algorithm, the real-time speech data is segmented into multiple spoken voices; Each spoken voice is input into a trained emotion recognition model to obtain the output voice emotion; Based on the voice emotions corresponding to all the spoken voices, the voice emotion features corresponding to the real-time voice data are determined.

[0014] As an optional implementation, in a second aspect of the present invention, the specific method by which the recognition module segments multiple spoken voices corresponding to the real-time speech data based on a speech segmentation algorithm includes: Based on the speech frequency analysis algorithm, the speech frequency change information corresponding to the real-time speech data is determined; Based on the peak frequency recognition algorithm, multiple frequency peaks in the speech frequency change information are determined; Cluster all frequency points that are greater than the corresponding frequency threshold before and after each frequency peak point to obtain multiple frequency point sets; Each frequency point set corresponds to a set of speech data corresponding to all time points, which is then used to define a single speech voice.

[0015] As an optional implementation, in a second aspect of the invention, the specific method by which the recognition module determines the voice emotion features corresponding to the real-time voice data based on the voice emotions corresponding to all the spoken voices includes: Generate a graph data node corresponding to each spoken voice, and determine the voice emotion corresponding to the spoken voice as the node feature corresponding to the graph data node; Calculate the time difference between any two of the spoken voices at any given time point; Calculate the speech similarity between any two of the spoken voices; The weighted sum of the time difference and the speech similarity is determined as the node association feature between the graph data nodes corresponding to the two spoken voices; All the graph data nodes and their corresponding node features and node association features are determined as the voice emotion features corresponding to the real-time voice data.

[0016] As an optional implementation, in a second aspect of the invention, the prediction module, based on a large language model, predicts the specific manner of the next adapted dialogue and its emotional tone according to the speech emotion features and the dialogue text, including: The dialogue text is input into the trained text direction prediction model to obtain the predicted text direction; The voice emotion features are input into a trained emotion description prediction graph neural network to obtain the output predicted emotion description. Based on the large language model, the next adapted dialogue and dialogue sentiment are predicted according to the predicted text direction and the predicted sentiment description.

[0017] As an optional implementation, in a second aspect of the invention, the prediction module, based on a large language model, predicts the specific manner of the next adapted dialogue and the dialogue sentiment according to the predicted text direction and the predicted sentiment description, including: Based on the preset prompt word engineering, corresponding prompt word text is assembled according to the predicted text direction and the predicted sentiment description; the prompt words are used to describe the strategy requirements for responding to users in the predicted sentiment description within the predicted text direction; The prompt text is input into the trained LLM model to obtain the next adapted dialogue and dialogue sentiment.

[0018] As an optional implementation, in a second aspect of the invention, the generation module generates the next outbound call voice corresponding to the outbound call user based on the next adapted dialogue and dialogue sentiment, using a speech generation algorithm, in a specific manner including: Determine the emotion tag corresponding to the dialogue emotion; The next adapted dialogue and the emotion tag are input into the trained speech generation algorithm to generate the next outbound call speech corresponding to the outbound call user.

[0019] As an optional implementation, in a second aspect of the invention, the specific method by which the generation module determines the emotion tag corresponding to the dialogue emotion includes: Determine whether the emotion in the dialogue has a matching emotion tag in the emotion tag library. If it does, directly identify it as the corresponding emotion tag. If it does not exist, calculate the emotion similarity between the template speech corresponding to each candidate emotion tag in the emotion tag library and the real-time speech data. The candidate emotion tag with the lowest emotion similarity is determined as the emotion tag corresponding to the dialogue emotion.

[0020] A third aspect of this invention discloses another large-model-driven emotion-smoothing evolution speech synthesis system, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the large model-driven emotion smooth evolution speech synthesis method disclosed in the first aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the large model-driven emotion smooth evolution speech synthesis method disclosed in the first aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention acquires real-time voice data and historical dialogue text of outbound call users, identifies voice emotion features based on a speech recognition model, predicts the next appropriate dialogue and dialogue emotion using a large language model, and generates the next outbound call voice based on a speech generation algorithm. This enables emotion-smooth evolution speech synthesis driven by a large model, supporting intelligent outbound call continuation, improving the coherence and user adaptability of outbound dialogue, and reducing the risk of communication interruption due to mismatch of emotions or intentions. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a large-model-driven emotion-smooth evolution speech synthesis method disclosed in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of a large model-driven emotion-smoothing speech synthesis system disclosed in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of another emotion-smooth evolution speech synthesis system based on a large model driven by an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] This invention discloses a method and system for emotion-smooth evolution speech synthesis based on a large model. By acquiring real-time voice data and historical dialogue text of outbound call users, it identifies voice emotion features based on a speech recognition model, predicts the next suitable dialogue and its emotion using a large language model, and generates the next outbound call voice based on a speech generation algorithm. This enables emotion-smooth evolution speech synthesis based on a large model, supporting intelligent outbound call continuation, improving the coherence and user adaptability of outbound dialogues, and reducing the risk of communication interruption due to mismatched emotions or intentions. Detailed explanations follow.

[0031] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a large-model-driven emotion-smoothing speech synthesis method disclosed in an embodiment of the present invention. Figure 1 The described large-model-driven emotion-smoothing evolution speech synthesis method can be applied to data processing systems / data processing devices / data processing servers (including local processing servers or cloud processing servers). Figure 1 As shown, this large model-driven emotion-smoothing speech synthesis method can include the following operations: 101. Obtain real-time voice data sent by outbound users and corresponding historical dialogue text for the same time period.

[0032] Optionally, the outbound caller can be a potential high-value financial management customer, a user being contacted by a telecommunications operator, a person who needs to postpone repayment, a student who has booked an online education course, or a patient being followed up by smart healthcare. This invention does not limit the scope of the caller.

[0033] Optionally, the real-time voice data may be a G.711 waveform audio stream transmitted via a VoLTE network, an AMR-WB narrowband voice packet, an Opus audio frame sequence transmitted based on a WebRTC architecture, or a raw PCM data stream captured by a call center hardware switch; the present invention does not limit this.

[0034] Optionally, the dialogue text for this historical time period can be ASR text logs converted by the speech recognition engine in previous multi-turn interactions, structured text context in multimodal conversation history, historical chat records stored in a distributed database, or text stream of the current call five minutes prior to the current call temporarily stored in a cache queue. This invention does not impose any limitations on these aspects.

[0035] 102. Based on the speech recognition model, identify the speech emotion features corresponding to real-time speech data.

[0036] Optionally, the speech recognition model can be a self-supervised audio representation model based on Wav2Vec 2.0, a Whisper end-to-end multi-task speech processing architecture, a Conformer speech recognition network, or an emotion feature extraction engine based on a deep residual network ResNet. This invention does not limit the specific model.

[0037] 103. Based on a large language model, predict the next adapted dialogue and dialogue emotion according to the speech emotion features and dialogue text.

[0038] Optionally, the large language model can be a large-scale autoregressive pre-trained language model with hundreds of billions of parameters, an LLaMA model fine-tuned by specific business instructions, a hybrid expert model MoE, or a dialogue model optimized for intelligent outbound calling scenarios. This invention does not limit the specific model.

[0039] 104. Based on the next adapted dialogue and dialogue sentiment, generate the next outbound call voice corresponding to the outbound call user using a voice generation algorithm.

[0040] Optionally, the next outbound voice message is sent to the outbound user to continue the conversation.

[0041] Optionally, the next adapted dialogue and dialogue emotion can be a generative text containing the next round of outbound call response strategy and a joint vector representation with emotion category scores, recommended dialogue that meets business compliance requirements, and corresponding emotion evolution tags. This invention does not limit these aspects.

[0042] Optionally, the speech generation algorithm can be a DiffWave generation network based on a diffusion model, a VITS end-to-end speech synthesis architecture based on a neural network, a deep synthesis system cascaded with Tacotron2 and WaveGlow, or an autoregressive audio generation model. This invention does not limit the specific implementation.

[0043] As can be seen, the above-mentioned embodiments of the invention acquire real-time voice data and historical dialogue text of outbound call users, identify voice emotion features based on a speech recognition model, predict the next suitable dialogue and dialogue emotion using a large language model, and generate the next outbound call voice based on a speech generation algorithm. This enables the realization of emotion-smooth evolution speech synthesis driven by a large model, thereby supporting intelligent outbound call continuation, improving the coherence and user adaptability of outbound call dialogues, and reducing the risk of communication interruption due to mismatch of emotions or intentions.

[0044] As an optional embodiment, the step above, identifying the voice emotion features corresponding to real-time voice data based on a speech recognition model, includes: Based on speech segmentation algorithms, multiple spoken voices corresponding to real-time speech data are segmented. Each spoken voice is input into a trained emotion recognition model to obtain the output voice emotion; Based on the emotional tone of all spoken words, the emotional tone features of real-time speech data are determined.

[0045] Optionally, the speech segmentation algorithm can be an endpoint detection (VAD) algorithm based on a long short-term memory network (LSTM), an audio segmentation algorithm based on a Gaussian mixture model (GMM), or a sliding window segmenter based on dynamic dual threshold determination of energy and zero-crossing rate. This invention does not limit the algorithm.

[0046] Optionally, the emotion recognition model can be an end-to-end three-dimensional convolutional neural network (3D-CNN), an emotion feature classifier based on a temporal convolutional network (TCN), or an acoustic multi-task learning network with a cross-attention mechanism; this invention does not limit the specific model.

[0047] As can be seen, through the above optional embodiments, multiple spoken voices are obtained by segmenting real-time speech data based on speech segmentation algorithms, and each spoken voice is input into a trained emotion recognition model to obtain the voice emotion. Based on all emotions, the voice emotion features are determined, thereby achieving fine segmentation of speech data and accurate emotion recognition, improving the accuracy and completeness of emotion feature extraction, and reducing the risk of emotion judgment bias caused by continuous unsegmented speech.

[0048] As an optional embodiment, the step above, segmenting multiple spoken voices corresponding to real-time speech data based on a speech segmentation algorithm, includes: Based on the speech frequency analysis algorithm, determine the speech frequency change information corresponding to real-time speech data; Based on the peak frequency recognition algorithm, multiple frequency peaks in speech frequency change information are identified. Cluster all frequency points that are greater than the corresponding frequency threshold before and after each frequency peak point to obtain multiple frequency point sets; Each frequency point set is used to identify a single speech by analyzing the speech data corresponding to all time points.

[0049] Optionally, the speech frequency analysis algorithm can be a short-time four-factor transform (STFT) algorithm based on fast four-factor transform (FFT), a continuous wavelet transform (CWT) algorithm, or a pitch period extraction algorithm based on autocorrelation function; this invention does not limit the algorithm.

[0050] Optionally, the frequency threshold can be an adaptive average noise power threshold, a frequency cutoff decibel value dynamically adjusted based on the current voice signal-to-noise ratio (SNR), or the mean plus standard deviation of the frequency envelope of historical silent segments; this invention does not impose any limitations.

[0051] Optionally, the clustering can be density-based DBSCAN clustering algorithm, one-dimensional K-Means clustering, hierarchical clustering algorithm, or local neighborhood aggregation based on dynamic time warping (DTW) distance; this invention does not limit the specific clustering method.

[0052] As can be seen, through the above optional embodiments, frequency peak points are obtained by analyzing speech frequency and identifying peaks, and high-frequency points before and after the peaks are clustered to obtain a set of frequency points and determine the spoken speech. This achieves speech segmentation based on frequency features, improves the accuracy and robustness of spoken speech segmentation, and reduces the risk of segmentation errors caused by traditional segmentation methods.

[0053] As an optional embodiment, the step above, determining the voice emotion features corresponding to the real-time voice data based on the voice emotions corresponding to all spoken voices, includes: Generate graph data nodes corresponding to each spoken speech, and determine the speech emotion corresponding to the spoken speech as the node feature corresponding to the graph data node; Calculate the time difference between any two spoken audio points; Calculate the speech similarity between any two spoken words; The weighted sum of time difference and speech similarity is used to determine the node association feature between the graph data nodes corresponding to the two spoken voices; All graph data nodes, their corresponding node features, and node association features are identified as the voice emotion features corresponding to the real-time voice data.

[0054] Optionally, the graph data node can be a vertex representation object in a graph neural network (GNN), a nonlinear associated node entity stored in a graph database system, or a temporal feature carrier representing a single breath or speech cycle; this invention does not impose any limitations.

[0055] Optionally, the speech similarity can be the cosine similarity, Pearson correlation coefficient, or structural similarity index (SSIM) between the spectrograms of two speech segments extracted using a pre-trained audio encoder in the latent space; this invention does not limit the specific parameters.

[0056] As can be seen, through the above optional embodiments, by generating graph data nodes for each spoken speech and using speech emotion as node features, calculating the weighted sum of time difference and speech similarity as node association features to construct speech emotion features, a graph structured representation of speech emotion is realized, improving the temporal association ability and expression depth of emotion features, and reducing the risk of missing emotion associations due to linear analysis.

[0057] As an optional embodiment, the above steps, including predicting the next adapted dialogue and dialogue emotion based on a large language model, according to speech emotion features and dialogue text, include: Input the dialogue text into the trained text direction prediction model to obtain the predicted text direction; The speech emotion features are input into a trained emotion description prediction graph neural network to obtain the output predicted emotion description. Based on a large language model, the next adapted dialogue and dialogue sentiment are predicted according to the predicted text direction and predicted sentiment description.

[0058] Optionally, the text direction prediction model can be a context intent classification network based on the BERT architecture, a conditional generation direction predictor based on XLNet, or a speech state transition determination engine integrated with Markov chains; this invention does not limit the specific model.

[0059] Optionally, the predicted text direction can be a predicted user's upcoming complaint and rights protection branch, a business subscription intention confirmation branch, a refusal to unsubscribe and response branch, or a routine question consultation branch; this invention does not limit this.

[0060] Optionally, the emotion description prediction graph neural network can be a graph spatial attention network (GAT), a graph spatial topological convolutional network (GCN), or a heterogeneous graph neural network (HGN), and this invention does not limit it.

[0061] As can be seen, through the above optional embodiments, the predicted text direction is obtained by inputting the dialogue text into the text direction prediction model, the predicted emotion description is obtained by inputting the voice emotion features into the emotion description prediction graph neural network, and the next suitable dialogue and dialogue emotion are predicted based on the large language model. This achieves joint intelligent prediction of dialogue direction and emotion, improves the rationality and naturalness of the continuation content, and reduces the risk of dialogue mismatch caused by single-dimensional prediction.

[0062] As an optional embodiment, the above steps, including predicting the next adapted dialogue and dialogue sentiment based on a large language model, according to the predicted text direction and predicted sentiment description, include: Based on the preset prompt word engineering, the corresponding prompt word text is assembled according to the predicted text direction and predicted sentiment description; The prompt text is input into the trained LLM model to obtain the next adapted dialogue and dialogue sentiment.

[0063] Optionally, cue words are used to describe the strategy requirements for responding to users in the predicted sentiment description within the predicted text direction.

[0064] Optionally, the prompt word engineering can be a templated structural engineering framework that includes role setting, context constraints, dynamic sample prompts (Few-Shot), and progressive reasoning (CoT), and this invention does not limit it.

[0065] Optionally, the prompt text can be dynamically concatenated XML structured text, a long text string in Markdown format encapsulated with specific tags, or a low-level system prompt stream specifically used for large language model inference; this invention does not impose any limitations.

[0066] As can be seen, through the above optional embodiments, by assembling the predicted text direction and predicted sentiment description based on preset prompt words, and inputting them into the trained LLM model, the next adapted dialogue and dialogue sentiment are obtained, thereby realizing the accurate generation of prompt-driven large language models, improving the strategic nature and adaptability of dialogue content generation, and reducing the risk of generation generalization caused by insufficient prompts.

[0067] As an optional embodiment, the step above, generating the next outbound call voice for the outbound user based on the next adapted dialogue and dialogue sentiment using a speech generation algorithm, includes: Determine the emotion label corresponding to the emotions in the conversation; The next matching dialogue and emotion label are input into the trained speech generation algorithm to generate the next outbound call speech for the outbound call user.

[0068] Optionally, the emotion label can be a discrete dimension classification label such as "high anger, mild anxiety, calm and rational, joy and welcome", or a VA dual-axis continuous value coordinate pair composed of valence and arousal. This invention does not limit the scope of the label.

[0069] As can be seen, through the above optional embodiments, by determining the emotion tag corresponding to the dialogue emotion, the next adapted dialogue and emotion tag are input into the trained speech generation algorithm to generate the next outbound call speech, realizing emotion tag-driven speech synthesis, improving the emotion matching degree and naturalness of the generated speech, and reducing the risk of speech emotion deviation caused by the lack of emotion tags.

[0070] As an optional embodiment, the step of determining the emotion tag corresponding to the emotion in the dialogue in the above steps includes: Determine if the emotion in the dialogue exists in the emotion tag library. If it does, directly identify it as the corresponding emotion tag. If it does not exist, calculate the emotion similarity between the template speech and the real-time speech data corresponding to each candidate emotion tag in the emotion tag library; The candidate emotion label with the lowest emotion similarity is determined as the emotion label corresponding to the emotion in the dialogue.

[0071] Optionally, the emotion tag library can be a standard emotion ontology knowledge graph embedded in a cache, a one-dimensional enumerated emotion classification dictionary stored in a local configuration table, or a feature vector index library containing multimodal acoustic emotion benchmark values. This invention does not limit the scope of the invention.

[0072] Optionally, the emotional similarity can be the KL divergence (Kullback-Leibler Divergence), relative entropy distance, or the correlation degree of feature vectors calculated by a contrastive learning network between the two audio segments in a specific emotional discrimination space. This invention does not limit the specific features.

[0073] As can be seen, through the above optional embodiments, by determining whether there is a matching tag for the dialogue emotion in the emotion tag library, if there is no matching tag, the emotion similarity with the template speech is calculated and the candidate tag with the lowest similarity is selected to determine the emotion tag, thereby realizing intelligent matching of emotion tags based on similarity optimization, improving the flexibility and accuracy of emotion tag determination, and reducing the risk of emotion labeling errors caused by incomplete tag library.

[0074] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a large-model-driven emotion-smoothing speech synthesis system disclosed in an embodiment of the present invention. Figure 2The described large-model-driven emotion-smoothing evolution speech synthesis system can be applied to data processing systems / data processing devices / data processing servers (including local processing servers or cloud processing servers). For example... Figure 2 As shown, this large model-driven emotion-smoothing evolution speech synthesis system may include: The acquisition module 201 is used to acquire real-time voice data sent by outbound call users and the corresponding historical time period dialogue text.

[0075] The recognition module 202 is used to identify the emotional features of real-time speech data based on a speech recognition model.

[0076] Prediction module 203 is used to predict the next adapted dialogue and dialogue emotion based on the large language model, according to the speech emotion features and dialogue text.

[0077] The generation module 204 is used to generate the next outbound call voice for the outbound call user based on the next adapted dialogue and dialogue emotion, using a voice generation algorithm.

[0078] Optionally, the next outbound voice message is sent to the outbound user to continue the conversation.

[0079] As can be seen, the above-mentioned embodiments of the invention acquire real-time voice data and historical dialogue text of outbound call users, identify voice emotion features based on a speech recognition model, predict the next suitable dialogue and dialogue emotion using a large language model, and generate the next outbound call voice based on a speech generation algorithm. This enables the realization of emotion-smooth evolution speech synthesis driven by a large model, thereby supporting intelligent outbound call continuation, improving the coherence and user adaptability of outbound call dialogues, and reducing the risk of communication interruption due to mismatch of emotions or intentions.

[0080] As an optional embodiment, the recognition module, based on a speech recognition model, identifies the specific methods by which it identifies the emotional features of real-time speech data, including: Based on speech segmentation algorithms, multiple spoken voices corresponding to real-time speech data are segmented. Each spoken voice is input into a trained emotion recognition model to obtain the output voice emotion; Based on the emotional tone of all spoken words, the emotional tone features of real-time speech data are determined.

[0081] As can be seen, through the above optional embodiments, multiple spoken voices are obtained by segmenting real-time speech data based on speech segmentation algorithms, and each spoken voice is input into a trained emotion recognition model to obtain the voice emotion. Based on all emotions, the voice emotion features are determined, thereby achieving fine segmentation of speech data and accurate emotion recognition, improving the accuracy and completeness of emotion feature extraction, and reducing the risk of emotion judgment bias caused by continuous unsegmented speech.

[0082] As an optional embodiment, the recognition module, based on a speech segmentation algorithm, segments multiple spoken voices corresponding to real-time speech data in the following specific ways: Based on the speech frequency analysis algorithm, determine the speech frequency change information corresponding to real-time speech data; Based on the peak frequency recognition algorithm, multiple frequency peaks in speech frequency change information are identified. Cluster all frequency points that are greater than the corresponding frequency threshold before and after each frequency peak point to obtain multiple frequency point sets; Each frequency point set is used to identify a single speech by analyzing the speech data corresponding to all time points.

[0083] As can be seen, through the above optional embodiments, frequency peak points are obtained by analyzing speech frequency and identifying peaks, and high-frequency points before and after the peaks are clustered to obtain a set of frequency points and determine the spoken speech. This achieves speech segmentation based on frequency features, improves the accuracy and robustness of spoken speech segmentation, and reduces the risk of segmentation errors caused by traditional segmentation methods.

[0084] As an optional embodiment, the recognition module determines the specific method of voice emotion features corresponding to real-time voice data based on the voice emotions corresponding to all spoken voices, including: Generate graph data nodes corresponding to each spoken speech, and determine the speech emotion corresponding to the spoken speech as the node feature corresponding to the graph data node; Calculate the time difference between any two spoken audio points; Calculate the speech similarity between any two spoken words; The weighted sum of time difference and speech similarity is used to determine the node association feature between the graph data nodes corresponding to the two spoken voices; All graph data nodes, their corresponding node features, and node association features are identified as the voice emotion features corresponding to the real-time voice data.

[0085] As can be seen, through the above optional embodiments, by generating graph data nodes for each spoken speech and using speech emotion as node features, calculating the weighted sum of time difference and speech similarity as node association features to construct speech emotion features, a graph structured representation of speech emotion is realized, improving the temporal association ability and expression depth of emotion features, and reducing the risk of missing emotion associations due to linear analysis.

[0086] As an optional implementation, the prediction module, based on a large language model, predicts the specific manner of the next adapted dialogue and its emotional tone based on speech emotion features and the dialogue text, including: Input the dialogue text into the trained text direction prediction model to obtain the predicted text direction; The speech emotion features are input into a trained emotion description prediction graph neural network to obtain the output predicted emotion description. Based on a large language model, the next adapted dialogue and dialogue sentiment are predicted according to the predicted text direction and predicted sentiment description.

[0087] As can be seen, through the above optional embodiments, the predicted text direction is obtained by inputting the dialogue text into the text direction prediction model, the predicted emotion description is obtained by inputting the voice emotion features into the emotion description prediction graph neural network, and the next suitable dialogue and dialogue emotion are predicted based on the large language model. This achieves joint intelligent prediction of dialogue direction and emotion, improves the rationality and naturalness of the continuation content, and reduces the risk of dialogue mismatch caused by single-dimensional prediction.

[0088] As an optional implementation, the prediction module, based on a large language model, predicts the specific manner of the next adapted dialogue and the dialogue sentiment based on the predicted text direction and predicted sentiment description, including: Based on the preset prompt word engineering, the corresponding prompt word text is assembled according to the predicted text direction and predicted sentiment description; optionally, the prompt words are used to describe the strategy requirements for responding to users in the predicted sentiment description within the predicted text direction. The prompt text is input into the trained LLM model to obtain the next adapted dialogue and dialogue sentiment.

[0089] As can be seen, through the above optional embodiments, by assembling the predicted text direction and predicted sentiment description based on preset prompt words, and inputting them into the trained LLM model, the next adapted dialogue and dialogue sentiment are obtained, thereby realizing the accurate generation of prompt-driven large language models, improving the strategic nature and adaptability of dialogue content generation, and reducing the risk of generation generalization caused by insufficient prompts.

[0090] As an optional embodiment, the generation module generates the next outbound call voice for the outbound user based on the next adapted dialogue and dialogue sentiment, using a speech generation algorithm, including: Determine the emotion label corresponding to the emotions in the conversation; The next matching dialogue and emotion label are input into the trained speech generation algorithm to generate the next outbound call speech for the outbound call user.

[0091] As can be seen, through the above optional embodiments, by determining the emotion tag corresponding to the dialogue emotion, the next adapted dialogue and emotion tag are input into the trained speech generation algorithm to generate the next outbound call speech, realizing emotion tag-driven speech synthesis, improving the emotion matching degree and naturalness of the generated speech, and reducing the risk of speech emotion deviation caused by the lack of emotion tags.

[0092] As an optional embodiment, the generation module determines the specific method for the emotion tag corresponding to the dialogue emotion, including: Determine if the emotion in the dialogue exists in the emotion tag library. If it does, directly identify it as the corresponding emotion tag. If it does not exist, calculate the emotion similarity between the template speech and the real-time speech data corresponding to each candidate emotion tag in the emotion tag library; The candidate emotion label with the lowest emotion similarity is determined as the emotion label corresponding to the emotion in the dialogue.

[0093] As can be seen, through the above optional embodiments, by determining whether there is a matching tag for the dialogue emotion in the emotion tag library, if there is no matching tag, the emotion similarity with the template speech is calculated and the candidate tag with the lowest similarity is selected to determine the emotion tag, thereby realizing intelligent matching of emotion tags based on similarity optimization, improving the flexibility and accuracy of emotion tag determination, and reducing the risk of emotion labeling errors caused by incomplete tag library.

[0094] Example 3 Please see Figure 3 , Figure 3 This is another emotion-smooth evolution speech synthesis system based on a large model driven by the present invention. Figure 3 The described large-model-driven emotion-smoothing evolution speech synthesis system is applied in data processing systems / data processing devices / data processing servers (wherein, the server includes local processing servers or cloud processing servers). Figure 3 As shown, this large model-driven emotion-smoothing evolution speech synthesis system may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the large model-driven emotion smooth evolution speech synthesis method described in Embodiment 1.

[0095] Example 4 This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps of the large model-driven emotion smooth evolution speech synthesis method described in Embodiment 1.

[0096] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the large model-driven emotion smooth evolution speech synthesis method described in Embodiment 1.

[0097] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0099] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0100] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0105] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0106] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0107] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0110] Finally, it should be noted that the method and system for emotion-smooth evolution speech synthesis based on large model-driven approach disclosed in the embodiments of this invention are merely preferred embodiments of the invention and are only used to illustrate the technical solutions of the invention, not to limit it. Although the invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention.

Claims

1. A method for emotion-smooth evolution speech synthesis based on a large model, characterized in that, The method includes: Obtain real-time voice data sent by outbound users and corresponding historical dialogue text for the same time period; Based on the speech recognition model, the speech emotion features corresponding to the real-time speech data are identified. Based on the large language model, the next adapted dialogue and dialogue emotion are predicted according to the voice emotion features and the dialogue text. Based on the next adapted dialogue and dialogue sentiment, a next outbound call voice is generated for the outbound user using a voice generation algorithm; the next outbound call voice is sent to the outbound user to continue the dialogue.

2. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 1, characterized in that, The process of identifying the emotional features of the real-time speech data based on the speech recognition model includes: Based on a speech segmentation algorithm, the real-time speech data is segmented into multiple spoken voices; Each spoken voice is input into a trained emotion recognition model to obtain the output voice emotion; Based on the voice emotions corresponding to all the spoken voices, the voice emotion features corresponding to the real-time voice data are determined.

3. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 2, characterized in that, The segmentation of multiple spoken voices corresponding to the real-time speech data based on the speech segmentation algorithm includes: Based on the speech frequency analysis algorithm, the speech frequency change information corresponding to the real-time speech data is determined; Based on the peak frequency recognition algorithm, multiple frequency peaks in the speech frequency change information are determined; Cluster all frequency points that are greater than the corresponding frequency threshold before and after each frequency peak point to obtain multiple frequency point sets; Each frequency point set corresponds to a set of speech data corresponding to all time points, which is then used to define a single speech voice.

4. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 2, characterized in that, The step of determining the voice emotion features corresponding to the real-time voice data based on the voice emotions corresponding to all the spoken voices includes: Generate a graph data node corresponding to each spoken voice, and determine the voice emotion corresponding to the spoken voice as the node feature corresponding to the graph data node; Calculate the time difference between any two of the spoken voices at any given time point; Calculate the speech similarity between any two of the spoken voices; The weighted sum of the time difference and the speech similarity is determined as the node association feature between the graph data nodes corresponding to the two spoken voices; All the graph data nodes and their corresponding node features and node association features are determined as the voice emotion features corresponding to the real-time voice data.

5. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 1, characterized in that, The method of predicting the next adapted dialogue and dialogue emotion based on the large language model, according to the speech emotion features and the dialogue text, includes: The dialogue text is input into the trained text direction prediction model to obtain the predicted text direction; The voice emotion features are input into a trained emotion description prediction graph neural network to obtain the output predicted emotion description. Based on the large language model, the next adapted dialogue and dialogue sentiment are predicted according to the predicted text direction and the predicted sentiment description.

6. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 5, characterized in that, The method based on a large language model, predicting the next adapted dialogue and dialogue sentiment according to the predicted text direction and the predicted sentiment description, includes: Based on the preset prompt word engineering, corresponding prompt word text is assembled according to the predicted text direction and the predicted sentiment description; the prompt words are used to describe the strategy requirements for responding to users in the predicted sentiment description within the predicted text direction; The prompt text is input into the trained LLM model to obtain the next adapted dialogue and dialogue sentiment.

7. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 1, characterized in that, The step of generating the next outbound call voice for the outbound call user based on the next adapted dialogue and dialogue emotion, using a speech generation algorithm, includes: Determine the emotion tag corresponding to the dialogue emotion; The next adapted dialogue and the emotion tag are input into the trained speech generation algorithm to generate the next outbound call speech corresponding to the outbound call user.

8. The emotion-smooth evolution speech synthesis method based on a large model as described in claim 7, characterized in that, Determining the emotion tag corresponding to the dialogue emotion includes: Determine whether the emotion in the dialogue has a matching emotion tag in the emotion tag library. If it does, directly identify it as the corresponding emotion tag. If it does not exist, calculate the emotion similarity between the template speech corresponding to each candidate emotion tag in the emotion tag library and the real-time speech data. The candidate emotion tag with the lowest emotion similarity is determined as the emotion tag corresponding to the dialogue emotion.

9. A large-model-driven emotion-smooth evolution speech synthesis system, characterized in that, The system includes: The acquisition module is used to acquire real-time voice data sent by outbound call users and the corresponding historical dialogue text for the same time period. The recognition module is used to identify the voice emotion features corresponding to the real-time voice data based on the voice recognition model; The prediction module is used to predict the next adapted dialogue and dialogue emotion based on the speech emotion features and the dialogue text, using a large language model. The generation module is used to generate the next outbound call voice corresponding to the outbound call user based on the next adapted dialogue and dialogue emotion, using a voice generation algorithm; the next outbound call voice is used to send to the outbound call user to continue the dialogue.

10. A large-model-driven emotion-smooth evolution speech synthesis system, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the large model-driven emotion smooth evolution speech synthesis method as described in any one of claims 1-8.

Images