Personalized text-to-speech method and system based on multi-emotion speech prototype and large model semantic segmentation control

Abstract

The application discloses a personalized text-to-speech method and system based on multi-emotion speech prototype and large model semantic segmentation control, which cuts the target person's voice, extracts and clusters emotion features, and constructs a multi-emotion speech prototype library; a large language model is used for semantic segmentation, emotion prediction and tone word insertion of the text to be synthesized, and each sentence emotion is mapped to the corresponding emotion prototype to drive the non-self-recurrent flow matching TTS model to generate speech. The scheme can realize multi-emotion, fine-grained and colloquial speech synthesis under the same timbre, and improve the naturalness of the speech and the emotional expression ability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and speech processing technology, specifically involving a personalized speech synthesis method and system that combines speech sentiment analysis, text semantic understanding and non-autoregressive text-to-speech (TTS). In particular, it utilizes a large language model to perform sentiment analysis, natural semantic segmentation and generation of interjections on text, and drives a multi-emotion speech prototype library to perform emotion-controlled personalized speech synthesis technology. Background Technology

[0002] With the development of deep learning and large models, TTS can synthesize speech with a timbre close to the target speaker and a high degree of naturalness, given a small amount of audio cues. Current mainstream technologies include: (1) speech generation models based on autoregressive structures, which predict speech representation frame by frame or token by token; and (2) non-autoregressive TTS models based on diffusion models or flow matching, which achieve high-quality and high-speed speech synthesis through continuous transformation from noise to speech representation.

[0003] Existing literature proposes a class of non-autoregressive TTS systems based on conditional flow matching and diffusion Transformer: during training, the input text is converted into a character sequence and expanded to the same length as the speech spectrogram using padding symbols. Then, under the condition of combining text with masked speech, flow matching is performed on the noise spectrogram for training. During inference, starting from the noise, an ordinary differential equation solver is used to gradually reconstruct the speech spectrogram. This type of method does not rely on explicit phoneme alignment and duration models, has fast inference speed, and possesses good zero-shot multi-speaker synthesis capabilities.

[0004] In terms of speech emotion, traditional emotion-based TTS often employs the following methods: labeling an entire sentence or segment of speech with a global emotion tag (such as happy, sad, calm, etc.) and controlling its generation through emotion embedding; or manually selecting several reference voices and directly inputting them as style conditions into the TTS model. These methods generally can only control emotion at the whole sentence or segment scale, making it difficult to achieve fine-grained emotion changes within a sentence based on text semantics, and lacking the ability to automatically model individual speaking habits and multiple emotional styles of a specific person.

[0005] On the other hand, existing TTS systems typically rely on simple punctuation and sentence segmentation or rule-based preprocessing for text processing. They rarely utilize large-scale language models for deep semantic understanding, natural sentence segmentation, or automatic insertion of interjections that match the character's speaking habits. This results in synthesized speech often having a reading-aloud style, lacking a sense of dialogue and the spoken details of a real human voice. Existing non-autoregressive TTS literature based on stream matching and diffusion Transformer also clearly points out that although the models perform well in zero-sample timbre imitation, they still lack fine-grained control over paralinguistic details such as emotion.

[0006] The closest existing technology to this invention is: (1) a non-autoregressive TTS model based on conditional flow matching: input character-level text sequence and Mel spectrogram corresponding to audio prompt speech; the model adopts a diffusion Transformer (DiT) structure to perform conditional flow matching training on noise spectrogram, and generates target spectrogram in conjunction with ODE solver during inference. (2) a TTS model with simple emotion control: by manual annotation or simple emotion classifier, several emotion labels are assigned to the training speech; during inference, specific emotion labels or reference speech are input to generate the corresponding whole sentence speech; text segmentation and tone details are usually determined by rules or implicitly learned by the model.

[0007] The shortcomings of existing technical solutions and the reasons for these shortcomings include the following:

[0008] (1) Lack of a mechanism to automatically extract multiple emotional archetypes from the target person’s voice. Existing TTS mostly only provides speaker control at the timbre level, and the emotion end often only provides a few global labels, which are usually general speakers rather than the multiple emotional styles of a specific person. It is impossible to automatically summarize stable personalized emotional archetypes such as anger, joy, and calmness from multiple segments of the target person’s voice.

[0009] (2) It is impossible to achieve dynamic control of intra-sentence emotions based on text semantics. Although existing non-autoregressive TTS based on stream matching is more natural in terms of rhythm, it has very limited control over emotions during inference. It generally only controls emotions through fixed voice prompts or whole-sentence emotion embedding, and cannot assign different emotions to different semantic segments in a sentence. Therefore, there is insufficient emotional fluctuation in long texts.

[0010] (3) The text processing is too crude, lacking natural semantic segmentation and automatic generation of modal particles. Most systems still rely on punctuation or simple rules for text preprocessing, without using large language models to perform natural segmentation from a semantic perspective, and without automatically inserting modal particles according to the context, making the generated speech more like reading aloud than dialogue.

[0011] (4) Emotional control is disconnected from the speaker's personal style. Even if emotional labels exist, they are often a uniform set of general emotions, rather than personal emotional prototypes automatically extracted from the historical speech of a particular target person. This results in large differences in the listening experience of the same emotional label between different speakers; it is difficult to reproduce the unique tone words, speaking speed, and rhythmic habits of the target person under different emotions.

[0012] (5) Existing non-autoregressive TTS systems do not adequately support fine-grained paralinguistic control. Although existing stream-matching-based TTS systems have achieved a simple text input-speech output process, the literature also points out that one of their limitations is that it is difficult to accurately control paralinguistic attributes, including emotions, and they can only rely on speech prompts and implicit learning to reflect a certain style.

[0013] In summary, a new technological solution is needed that can automatically extract multiple emotional archetypes from the target person's speech, utilize a large model for sentiment analysis and semantic segmentation of the text, combine textual emotions with speech emotional archetypes, and drive a non-autoregressive TTS model to generate multi-emotional, natural, and conversational speech. This solution addresses the following technical issues:

[0014] (1) The problem of multi-emotion modeling of target person. In the absence of a large number of manual emotion annotations, how to automatically extract several representative emotion speech prototypes based solely on the historical speech data of the target person, and build an emotion prototype library that can be used for TTS control.

[0015] (2) Text-driven fine-grained emotion control problem. How to use large language models to perform semantic understanding of text, so as to perform natural sentence segmentation and semantic segmentation of long texts; predict the appropriate emotion category for each semantic segment; map these emotion categories to the emotional prototype of the target person, and realize the dynamic intra-sentence emotion control of using different emotional voices for different semantic segments.

[0016] (3) Colloquial expressions and the generation of interjections. How to automatically add interjections that conform to the speaking habits of the target person to the text through a large language model without destroying the original semantics, so that the final synthesized speech is closer to the style of real dialogue.

[0017] (4) Integrate the above capabilities while maintaining the simplicity of the model structure and the efficiency of inference. On the basis of the existing non-autoregressive flow matching TTS framework, introduce the emotion prototype library and the semantic control information of the large model, while maintaining the simplicity of the overall model architecture and the efficiency of inference. Summary of the Invention

[0018] Purpose of the invention: To address the above problems, this invention proposes a personalized text-to-speech method and system based on multi-emotional speech prototypes and large-scale model semantic segmentation control.

[0019] Technical solution: To achieve the objectives of this invention, the technical solution adopted is as follows:

[0020] A personalized text-to-speech method based on multi-emotional speech prototypes and large-model semantic segmentation control includes an offline stage, an online stage, and emotion mapping and TTS control.

[0021] In the offline stage, the voice of the target person is sliced ​​and the voice emotion vector is extracted. At the same time, semantic analysis is performed on the corresponding transcribed text. A joint representation space is constructed based on the voice emotion vector and the text semantic tags. In the joint representation space, a multi-emotion prototype library of the target person is formed by clustering.

[0022] During the online phase, a large language model is used to perform deep semantic analysis, complete text semantic segmentation and sentiment prediction, and transform the text from written language to spoken language.

[0023] Map the emotion label of each sentence to a certain emotion prototype in the emotion prototype library to obtain the corresponding emotion style vector and / or reference speech.

[0024] In the non-autoregressive stream matching TTS model, the emotion style vector is used as a conditional input to adjust the prosody, energy, fundamental frequency and speech rate of the synthesized speech, thereby achieving emotion changes within the sentence.

[0025] Specifically, the following steps are included:

[0026] S1: Collect raw audio data of the target person for a certain duration, preprocess the audio, and transcribe the audio into text to form audio-text pairs;

[0027] S2: The preprocessed speech sequence is sliced ​​into equal-length or adaptive segments within a certain time window to obtain several speech segments;

[0028] Acoustic features are extracted for each speech segment, and the acoustic features are input into a pre-trained emotion recognition network to output an initial emotion vector.

[0029] S3: Based on a large language model, text semantic analysis and natural segmentation are performed to construct character style and form a joint representation space of speech segments, emotion vectors, semantic tags and interjection information;

[0030] S4: Collect the emotion vectors and semantic labels of all speech segments, perform emotion prototype clustering, and build a multi-emotion speech prototype library;

[0031] S5: Train a non-autoregressive flow matching TTS model driven by emotion prototype vectors, and add an emotion control dimension to the existing flow matching TTS to achieve multi-emotion and multi-style synthesis under the same timbre.

[0032] S6: Performs online text processing, semantic natural segmentation, and automatic insertion of modal particles to obtain an ordered list of sentences;

[0033] S7: For the emotion tag of each sentence, search in the emotion prototype library to obtain the emotion prototype vector and reference emotion speech segment corresponding to each sentence, so as to realize the mapping from text emotion to emotion prototype.

[0034] S8: Perform sentence-level emotion-controlled TTS synthesis on the colloquial text of each sentence to obtain sentence speech with corresponding emotions;

[0035] S9: Speech Segment Assembly and Prosody Smoothing: Assembles each emotional speech segment in sentence order; performs energy, fundamental frequency, and duration smoothing at the assembly boundaries, and outputs the final complete speech file to the user.

[0036] Furthermore, in S3, text semantic analysis and natural segmentation based on a large language model include:

[0037] For each speech segment, the corresponding text is input into a large language model for analysis. The model output includes: semantic sentence segmentation results and sentiment tendency labels for each sentence.

[0038] Automatically insert modal particles based on context; the insertion positions include the beginning of a clause, a short semantic pause, before or after a transitional or emphatic semantic unit, and the ending of a sentence;

[0039] The processed text is associated with the speech segments and the initial emotion vector to form a joint representation space of speech segments, emotion vectors, semantic tags, and interjection information.

[0040] Furthermore, the insertion of modal particles is carried out under the constraint of maintaining the semantics of the original text; after the large language model completes semantic segmentation, it is determined by combining the contextual semantics, sentence boundaries, semantic pauses, transition or emphasis positions, question or exclamation positions, as well as the target person's commonly used modal particles and sentence habits obtained from offline statistics.

[0041] Furthermore, in S4, a clustering algorithm is used to cluster speech segments in the joint representation space to obtain K emotion clusters;

[0042] Perform the following operations for each emotion cluster:

[0043] Calculate the cluster center emotion vector as the emotion prototype vector to characterize the typical acoustic features of this type of emotion;

[0044] Select several representative speech segments from the cluster and store them in the emotion prototype speech library;

[0045] The representative audio segments include:

[0046] Speech segments whose emotional features and model confidence scores are both above a threshold; speech segments that do not contain noise or anomalies; and speech segments whose average style is close to that emotion, i.e., whose vector distance is less than a certain threshold.

[0047] The modal particles and sentence structures within this cluster are statistically analyzed to serve as a reference for the colloquial style of subsequent texts.

[0048] The target person's multi-emotional speech prototype library is obtained, including: emotion category number, corresponding emotion prototype vector, corresponding representative speech segment, corresponding modal particles and pragmatic features.

[0049] Furthermore, the design of the training and emotion control interface for the non-autoregressive flow matching TTS model in S5 includes:

[0050] A non-autoregressive TTS model based on conditional flow matching and diffusion Transformer is used as the basic structure for speech generation to generate conversational text that fits the speaking habits of the target person.

[0051] The colloquial text, after character encoding and padding symbols, is expanded to have the same length as the Mel sound spectrum, thus obtaining text encoding features;

[0052] Obtain the corresponding emotion prototype vector and / or the emotion prototype spectrogram of representative emotion speech segments in the emotion prototype speech library.

[0053] The non-autoregressive stream matching TTS model takes text encoding and emotion prototype speech spectrum as conditional input information, and random noise speech spectrum as initial state to generate target speech spectrum; then it is restored to the target person's speech waveform through a vocoder.

[0054] Designing an emotion control interface in a non-autoregressive TTS model structure based on conditional flow matching and diffusion Transformer:

[0055] The emotion prototype vector and the target speaker vector are used together as conditional inputs into the normalization layer or attention module of the model.

[0056] During training, emotion classifiers or automatic annotation tools are used to construct emotion labels for the training speech, enabling the model to learn the mapping from changes in emotion vectors to changes in acoustic features.

[0057] Furthermore, in S6, online text processing, semantic natural segmentation, and automatic insertion of modal particles include:

[0058] The user inputs the text content to be synthesized;

[0059] A large language model is used for text input. The model completes the following tasks based on prompt words:

[0060] Segment the text naturally from a semantic perspective;

[0061] Based on the context, a preliminary sentiment prediction is given for each clause;

[0062] Based on the target person's usage habits of interjections in the offline phase statistics, interjections or phrases are inserted to make the sentences more conversational;

[0063] The output is an ordered list of sentences, including the colloquial text corresponding to each sentence and the sentiment tag for each sentence.

[0064] Furthermore, in S7, the mapping from textual emotion to emotion prototypes includes:

[0065] For each clause's sentiment tag, search the sentiment prototype library:

[0066] If the label corresponds one-to-one with the prototype category, the prototype is selected directly; if the output is a continuous emotion representation vector, the closest emotion prototype is selected by nearest neighbor search or similarity sorting.

[0067] The emotional prototype vector and reference emotional speech segment corresponding to each sentence are obtained.

[0068] Furthermore, in S8, sentence-level emotion-controlled TTS synthesis includes:

[0069] The colloquial text of each clause is converted into a character sequence and length-paded; the emotional prototype vector and the target speaker vector corresponding to the clause are used as conditional inputs to the non-autoregressive TTS model.

[0070] The model employs conditional flow matching and an ODE solver to generate a target Mel spectrum from a randomly initialized noise spectrum; the Mel spectrum is then converted into waveform audio using a vocoder to obtain sentence-level speech with corresponding emotions.

[0071] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0072] (1) Automatically construct a multi-emotion prototype library from the target person's voice. This invention automatically models multiple emotions of the target person through 3-5 second segment-level emotion analysis and clustering, instead of using general emotion tags, so that emotion control is more in line with the real expression of the specific person.

[0073] (2) Using a large language model for semantic natural segmentation and sentiment prediction. Compared with simple sentence segmentation based on punctuation, this invention uses a large language model to segment text and determine sentiment at the semantic level, making pauses and sentiment changes more consistent with semantic logic.

[0074] (3) Automatic insertion of interjections to generate conversational text. This invention automatically supplements interjections on the text side and refers to the interjections and sentence patterns commonly used by the target character under different emotions to achieve personalization at the text level, significantly enhancing the conversational and character-like feel of the synthesized speech.

[0075] (4) The complete link from emotion prototype to TTS control. This invention achieves fine and controllable emotion control by designing “text emotion tag → emotion prototype library → emotion style vector → TTS conditional input”, while maintaining the simplicity and efficiency of the non-autoregressive TTS structure. Attached Figure Description

[0076] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0077] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0078] The personalized text-to-speech method based on multi-emotional speech prototypes and large-model semantic segmentation control described in this invention will be illustrated below with an example of a "multi-emotional TTS system for personalized audio content generation," but this invention is not limited to this scenario.

[0079] (1) Offline stage: Construct a multi-emotion prototype library for the target character.

[0080] The target person's voice is sliced ​​into 3-5 second segments, and each segment is analyzed jointly by an acoustic emotion recognition model and a large language model.

[0081] The large language model performs semantic understanding based on the corresponding transcribed text, achieving natural segmentation and completion of interjections, thereby obtaining text-semantic labels that are closer to actual speaking habits;

[0082] Segments are clustered in the joint space of speech emotion vectors and text semantic labels to obtain 5–8 emotion prototypes. Corresponding acoustic style vectors and representative speech segments are extracted to form an emotion prototype library. For each emotion category, the cluster center vector is calculated as the acoustic style vector, and the Top-K speech segments closest to the center are selected as representative speech segments to construct the emotion prototype library.

[0083] The speech emotion vector is a high-dimensional acoustic feature vector directly output by the Acoustic Emotion Recognition (SER) model. The model takes a 3-5 second speech segment of the target person as input and outputs the speech emotion vector.

[0084] (2) Online stage: text semantic segmentation and sentiment prediction.

[0085] For synthesized text, a large language model is used for deep semantic analysis: it automatically completes natural sentence segmentation instead of simple punctuation segmentation; it predicts the target emotion for each sentence, such as calm, joy, anger, surprise, etc.; and it automatically inserts interjections that conform to the spoken language habits of the characters, so that the text is transformed from written language into spoken language.

[0086] (3) Emotional mapping and TTS control.

[0087] Map the emotion label of each sentence to a certain emotion prototype in the emotion prototype library to obtain the corresponding emotion style vector and / or reference speech.

[0088] In the non-autoregressive stream matching TTS model, the emotion style vector is used as a conditional input to adjust the prosody, energy, fundamental frequency and speech rate of the synthesized speech, thereby achieving emotion changes within the sentence.

[0089] The generated speech maintains consistency with the target person in timbre; reflects emotional changes consistent with the text content in different semantic segments; and through the insertion of interjections and natural semantic segmentation, it makes the listening experience closer to real dialogue, storytelling, or character performance, significantly improving the ability to express emotions and the naturalness of the speech.

[0090] The embodiments of the present invention achieve the following effects:

[0091] (1) 3-5 second segment-level emotion prototype clustering: under limited speech samples, 5-8 representative emotional voices of the target person are automatically obtained, so that the subsequent synthesis can start from the "real speech prototype" to control emotions and improve the authenticity and consistency of emotional expression.

[0092] (2) Large model-driven text semantic segmentation and emotion labeling make the rhythmic information such as pauses, speech rate, and stress closer to the semantic structure, avoiding mechanical reading; at the same time, it realizes natural emotional fluctuations in long texts.

[0093] (3) Automatic insertion of interjections and colloquialization: Without destroying the original meaning, the written text is converted into colloquial text that is close to the speaking habits of people, making the synthesized speech more human and closer to the actual dialogue or storytelling scene.

[0094] (4) Emotion prototype vector-driven non-autoregressive TTS control adds an emotion control dimension on the basis of the high naturalness of existing flow matching TTS, realizes multi-emotion and multi-style synthesis under the same timbre, and supports intra-sentence emotion switching.

[0095] Multi-role emotion template extension: Extend the above method to multi-role scenarios, build emotion prototype libraries for different roles, and identify speaker tags in text through a large model to realize multi-role multi-emotion TTS in multi-person dialogue scenarios; it can be used in complex dialogue scenarios such as audio dramas and multi-role customer service robots.

[0096] Linking Business Strategy with Emotion Control: In scenarios such as customer service and education, business rules can be superimposed on text emotion prediction (e.g., a "soothing" emotion is preferred in complaint scenarios, and a "enthusiastic" emotion is preferred in promotional scenarios) to constrain or correct emotion labels; ensuring that the synthesized speech conforms to both the semantics of the text and the needs of business strategies in terms of emotion.

[0097] The personalized text-to-speech system of the present invention, based on multi-emotional speech prototypes and large-model semantic segmentation control, includes:

[0098] 1. The system architecture or scenario of the application.

[0099] The system adopts an architecture that combines cloud servers with terminal calls, including:

[0100] Voice Sample Acquisition and Management Subsystem: Used to collect, upload, and manage the historical voice data of the target person and its transcribed text;

[0101] Emotion Prototype Construction Subsystem (Offline Operation): Slices historical speech, performs emotion analysis and clustering, and constructs an emotion prototype library;

[0102] Text Semantic Analysis and Sentiment Prediction Subsystem: Based on a large language model, it performs semantic understanding, natural segmentation, interjection insertion, and sentiment prediction on the input text;

[0103] Emotion mapping and TTS synthesis subsystem: Introducing emotion prototypes and text emotion control information into a non-autoregressive stream matching TTS model to generate target speech;

[0104] The speech post-processing and playback subsystem is responsible for synthesizing segments, smoothing prosody, converting formats, and returning the data to the terminal.

[0105] The system can be deployed on a GPU server cluster, or it can be simplified and deployed on a local high-performance terminal device.

[0106] 2. The physical components involved.

[0107] The computing server (including CPU, GPU, memory, storage hard drive, etc.) is used for training and inference; the recording device / microphone or audio capture card is used to capture the voice of the target person; the network device is used for communication between the terminal and the server; the terminal device, such as PC, mobile phone or tablet, is used to upload voice, input text and play synthesized speech.

[0108] Specific implementation example 1:

[0109] S1: Voice acquisition and preprocessing of the target person.

[0110] Collect or import the raw voice data of the target person, which can range from several minutes to several hours; perform preprocessing on the voice such as noise reduction, coarse segmentation of silent segments, and uniform sampling rate; use an automatic speech recognition (ASR) model to transcribe the voice into text, forming voice segment-text pairs.

[0111] S2: Speech segmentation and initial emotion feature extraction.

[0112] The preprocessed speech sequence is sliced ​​into equal-length or adaptive segments in a 3–5 second window to obtain several speech segments; acoustic features such as Mel spectrum, fundamental frequency curve, energy envelope, and speech rate are extracted for each speech segment; the above acoustic features are input into a pre-trained emotion recognition network to output an initial emotion vector (e.g., a vector representation in a multidimensional continuous emotion space).

[0113] S3: Text semantic analysis and natural segmentation based on large language models (for constructing character styles).

[0114] For each speech segment, the corresponding text is input into a large language model for analysis. The model output includes: more granular semantic sentence segmentation results (such as splitting long sentences into multiple sentences or merging short sentences); and approximate sentiment labels for each sentence.

[0115] Automatically inserted interjections based on context, such as "um," "ah," "you know," and "actually," make the text more closely resemble the person's spoken expression style. The processed text is then associated with the speech segments and the initial emotion vector to form a joint representation of "speech segment - emotion vector - semantic tag - interjection information."

[0116] S4: Emotional prototype clustering and construction of a multi-emotional speech prototype library.

[0117] Collect the emotion vectors and semantic labels of all speech segments; use clustering algorithms (such as K-means, spectral clustering, etc.) to cluster the segments in the joint space to obtain K emotion clusters; for each emotion cluster: calculate the emotion vector of the cluster center as the emotion prototype vector, which is used to characterize the typical acoustic features of this type of emotion; select several representative speech segments in the cluster and store them in the emotion prototype speech library.

[0118] Representative speech segments include those with distinct emotional features and high model confidence; those free of noise / anomalies; and those closely resembling the average style of that emotion, i.e., with a vector distance less than a certain threshold. Common modal particles and sentence structures within this cluster are statistically analyzed as a style reference for subsequent text colloquialization.

[0119] The target person's multi-emotional speech prototype library is obtained, including: emotion category number; corresponding emotion prototype vector; corresponding representative speech segment; corresponding common modal particles and pragmatic features.

[0120] S5: Design of interface for training and emotion control of non-autoregressive flow matching TTS model.

[0121] A non-autoregressive TTS model based on conditional flow matching and diffusion Transformer is adopted as the basic structure for speech generation: the colloquial text that matches the speaking habits of the target person is encoded into characters, and the text encoding features are obtained by expanding it to the same length as the Mel spectrum by padding symbols; the model uses the text encoding features, the target person's speaker vector, and the emotion prototype vector and / or emotion prototype spectrum as conditional inputs to gradually generate the target Mel spectrum from the random noise spectrum, and restores it to the target person's speech waveform through a vocoder.

[0122] Design an emotion control interface within the architecture:

[0123] The emotion prototype vector and the target speaker vector are fed together as conditional inputs into the normalization layer or attention module of the model. During training, emotion classifiers or automatic annotation tools can be used to construct emotion labels for the training speech, so that the model learns the mapping of "changes in emotion vector → changes in acoustic features".

[0124] S6: Online text processing, natural semantic segmentation, and automatic insertion of modal particles.

[0125] The user inputs the text to be synthesized; the text is input into a large language model, which performs the following tasks based on prompts: naturally segments the text from a semantic perspective to avoid abnormal sentence breaks caused by punctuation errors or omissions; provides a preliminary sentiment prediction for each sentence based on the context; and inserts interjections or phrases at appropriate positions based on the target person's usage habits as statistically analyzed in the offline phase to make the sentences more colloquial.

[0126] Existing technologies may include methods for rewriting text into colloquial language and adding interjections based on rules or general language models. However, these methods are usually only geared towards general text processing and do not take into account the usage habits of interjections obtained from the target person's historical speech, sentence emotion tags, and multi-emotion speech prototype libraries. Furthermore, the results of interjection insertion are not used for subsequent personalized emotion-controlled TTS synthesis.

[0127] The insertion location can be specified as: at the beginning of a clause, at a semantic pause, before or after a transitional or emphatic semantic unit, at a question or exclamation, or at the end of a sentence where the tone is concluding. The judgment criteria include the semantic sentence segmentation results output by the large language model, the semantic relationships within the context, the clause sentiment labels, punctuation or pause information, and the target person's commonly used modal particles and sentence patterns under different sentiment clusters obtained from offline statistics. In actual output, the large language model can directly output the colloquial text after insertion, or it can output the insertion location index and the inserted modal particle; it can also perform semantic consistency checks between the inserted colloquial text and the original text to confirm that the insertion of modal particles and the colloquialization transformation have not changed the core semantics of the original text.

[0128] The output is an ordered list of sentences: the colloquial text corresponding to each sentence; and the emotion label for each sentence (e.g., calm, joy, anger, surprise, sadness, determination, etc.).

[0129] S7: Mapping from textual emotion to emotional prototype.

[0130] For each sentence's emotion tag, a search is performed in the emotion prototype library: if the tag corresponds one-to-one with the prototype category, the prototype is selected directly; if it is a continuous emotion space, the closest emotion prototype can be selected through nearest neighbor search or similarity sorting. This yields: an emotion prototype vector corresponding to each sentence; and optional reference emotion speech segments.

[0131] S8: Sentence-level emotion-controlled TTS synthesis.

[0132] The colloquial text of each sentence is converted into a character sequence and length-padded; the emotion prototype vector and the target speaker vector corresponding to the sentence are used as conditional inputs to a non-autoregressive TTS model; the model uses conditional flow matching and an ODE solver to gradually generate the target Mel spectrum from the noise spectrum; the Mel spectrum is converted into waveform audio by a vocoder to obtain the sentence speech with the corresponding emotion.

[0133] S9: Speech segment splicing and prosodic smoothing.

[0134] The emotional speech segments are spliced ​​together in sentence order; energy, fundamental frequency, and duration are smoothed at the splicing boundaries to avoid abrupt changes; the final complete speech file is then returned to the user.

Claims

1. A personalized text-to-speech method based on multi-emotion speech prototype and large model semantic segmentation control, characterized in that, This includes offline and online phases, as well as emotion mapping and TTS control; In the offline stage, the voice of the target person is sliced ​​and the voice emotion vector is extracted. At the same time, semantic analysis is performed on the corresponding transcribed text. A joint representation space is constructed based on the voice emotion vector and the text semantic tags. In the joint representation space, a multi-emotion prototype library of the target person is formed by clustering. During the online phase, a large language model is used to perform deep semantic analysis, complete text semantic segmentation and sentiment prediction, and transform the text from written language to spoken language. Map the emotion label of each sentence to a certain emotion prototype in the emotion prototype library to obtain the corresponding emotion style vector and / or reference speech. In the non-autoregressive stream matching TTS model, the emotion style vector is used as a conditional input to adjust the prosody, energy, fundamental frequency and speech rate of the synthesized speech, thereby achieving emotion changes within the sentence.

2. The method according to claim 1, characterized in that, Specifically, the following steps are included: S1: Collect raw audio data of the target person for a certain duration, preprocess the audio, and transcribe the audio into text to form audio-text pairs; S2: The preprocessed speech sequence is sliced ​​into equal-length or adaptive segments within a certain time window to obtain several speech segments; Acoustic features are extracted for each speech segment, and the acoustic features are input into a pre-trained emotion recognition network to output an initial emotion vector. S3: Based on a large language model, text semantic analysis and natural segmentation are performed to construct character style and form a joint representation space of speech segments, emotion vectors, semantic tags and interjection information; S4: Collect the emotion vectors and semantic labels of all speech segments, perform emotion prototype clustering, and build a multi-emotion speech prototype library; S5: Train a non-autoregressive flow matching TTS model driven by emotion prototype vectors, and add an emotion control dimension to the existing flow matching TTS to achieve multi-emotion and multi-style synthesis under the same timbre. S6: Performs online text processing, semantic natural segmentation, and automatic insertion of modal particles to obtain an ordered list of sentences; S7: For the emotion tag of each sentence, search in the emotion prototype library to obtain the emotion prototype vector and reference emotion speech segment corresponding to each sentence, so as to realize the mapping from text emotion to emotion prototype. S8: Perform sentence-level emotion-controlled TTS synthesis on the colloquial text of each sentence to obtain sentence speech with corresponding emotions; S9: Speech Segment Assembly and Prosodic Smoothing: Assembles each emotional speech segment according to the sentence order; Energy, fundamental frequency, and duration are smoothed at the splicing boundaries, and the final complete audio file is returned to the user.

3. The method according to claim 1, characterized in that, In S3, text semantic analysis and natural segmentation based on a large language model include: For each speech segment, the corresponding text is input into a large language model for analysis. The model output includes: semantic sentence segmentation results and sentiment tendency labels for each sentence. Automatically insert modal particles based on context; the insertion positions include the beginning of a clause, a short semantic pause, before or after a transitional or emphatic semantic unit, and the ending of a sentence; The processed text is associated with the speech segments and the initial emotion vector to form a joint representation space of speech segments, emotion vectors, semantic tags, and interjection information.

4. The method according to claim 3, characterized in that, The insertion of modal particles is carried out under the constraint of preserving the semantics of the original text; After semantic segmentation is completed by the large language model, the target person's commonly used interjections and sentence habits are determined by combining contextual semantics, sentence boundaries, semantic pauses, positions of transitions or emphasis, positions of interrogative or exclamatory tones, as well as statistics obtained in the offline stage.

5. The method according to claim 2, characterized in that, In S4, a clustering algorithm is used to cluster speech segments in the joint representation space to obtain K emotion clusters; Perform the following operations for each emotion cluster: Calculate the cluster center emotion vector as the emotion prototype vector to characterize the typical acoustic features of this type of emotion; Select several representative speech segments from the cluster and store them in the emotion prototype speech library; The representative audio segments include: Speech segments whose emotional features and model confidence scores are both above a threshold; speech segments that do not contain noise or anomalies; and speech segments whose average style is close to that emotion, i.e., whose vector distance is less than a certain threshold. The modal particles and sentence structures within this cluster are statistically analyzed to serve as a reference for the colloquial style of subsequent texts. The target person's multi-emotional speech prototype library is obtained, including: emotion category number, corresponding emotion prototype vector, corresponding representative speech segment, corresponding modal particles and pragmatic features.

6. The method according to claim 2, characterized in that, The design of the training and emotion control interface for the non-autoregressive flow matching TTS model in S5 includes: A non-autoregressive TTS model based on conditional flow matching and diffusion Transformer is used as the basic structure for speech generation to generate conversational text that fits the speaking habits of the target person. The colloquial text, after character encoding and padding symbols, is expanded to have the same length as the Mel sound spectrum, thus obtaining text encoding features; Obtain the corresponding emotion prototype vector and / or the emotion prototype spectrogram of representative emotion speech segments in the emotion prototype speech library. The non-autoregressive stream matching TTS model takes text encoding and emotion prototype speech spectrum as conditional input information, and random noise speech spectrum as initial state to generate target speech spectrum; then it is restored to the target person's speech waveform through a vocoder. Designing an emotion control interface in a non-autoregressive TTS model structure based on conditional flow matching and diffusion Transformer: The emotion prototype vector and the target speaker vector are used together as conditional inputs into the normalization layer or attention module of the model. During training, emotion classifiers or automatic annotation tools are used to construct emotion labels for the training speech, enabling the model to learn the mapping from changes in emotion vectors to changes in acoustic features.

7. The method according to claim 2, characterized in that, In S6, online text processing, semantic natural segmentation, and automatic insertion of modal particles include: The user inputs the text content to be synthesized; A large language model is used for text input. The model completes the following tasks based on prompt words: Segment the text naturally from a semantic perspective; Based on the context, a preliminary sentiment prediction is given for each clause; Based on the target person's usage habits of interjections in the offline phase statistics, interjections or phrases are inserted to make the sentences more conversational; The output is an ordered list of sentences, including the colloquial text corresponding to each sentence and the sentiment tag for each sentence.

8. The method according to claim 2, characterized in that, In S7, the mapping from textual emotion to emotion prototypes includes: For each clause's sentiment tag, search the sentiment prototype library: If the label corresponds one-to-one with the prototype category, the prototype is selected directly; if the output is a continuous emotion representation vector, the closest emotion prototype is selected by nearest neighbor search or similarity sorting. The emotional prototype vector and reference emotional speech segment corresponding to each sentence are obtained.

9. The method according to claim 2, characterized in that, In S8, sentence-level emotion-controlled TTS synthesis includes: The colloquial text of each clause is converted into a character sequence and length-paded; the emotional prototype vector and the target speaker vector corresponding to the clause are used as conditional inputs to the non-autoregressive TTS model. The model employs conditional flow matching and an ODE solver to generate a target Mel spectrum from a randomly initialized noise spectrum; the Mel spectrum is then converted into waveform audio using a vocoder to obtain sentence-level speech with corresponding emotions.

10. A personalized text-to-speech system based on multi-emotional speech prototypes and large-scale model semantic segmentation control, characterized in that: The system implements the personalized text-to-speech method based on multi-emotional speech prototypes and large-model semantic segmentation control as described in any one of claims 1-9.

Images