Audio TTS method for constructing multi-emotion and text semantic matching based on tone conversion

Abstract

The invention provides an audio TTS method for constructing multi-emotion and text semantic matching based on timbre conversion, and belongs to the technical field of audio processing, and the method comprises the steps: constructing a multi-emotion target timbre prompt wav set: collecting multi-emotion audios, training and finely tuning a target timbre conversion model, and converting the multi-emotion audios into multi-emotion prompt wav of a target timbre through the model; and TTS generation of text semantics and emotion matching: inputting a to-be-synthesized text, analyzing a text emotion category, calculating semantic similarity based on a text embedding model, screening adaptive prompt wav, and generating target voice through a TTS large model. According to the method, automatic batch generation of multi-emotion target timbre audios is realized, the manpower and time cost is reduced, the emotion and semantic suitability of TTS speech and text context is improved, and the speech synthesis quality is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to an audio TTS method based on timbre conversion to construct multi-emotion and text semantic matching. Background Technology

[0002] Text-to-speech (TTS) technology, as one of the core technologies of human-computer interaction, has been widely used in smart speakers, navigation systems, audio content production, and other fields. With the upgrading of user needs, higher requirements are being placed on the personalization of TTS voice (such as specific target timbre) and scene adaptability (such as matching the emotion of the text).

[0003] In existing technologies, acquiring TTS speech of a target voice under different emotions mainly relies on manually collecting multi-emotion audio of the target voice or hiring voice actors to record speaking audio of different emotions, and then training a TTS model based on these audio recordings. However, this method has significant drawbacks:

[0004] High cost: Manual collection or professional dubbing requires a lot of manpower, and the recording, screening and annotation of audio are time-consuming, resulting in high time and economic costs;

[0005] Incomplete emotional coverage: Manual methods cannot cover all common basic emotions, such as anger, sadness, and happiness, as well as advanced emotions, such as anxiety, sarcasm, and sincerity. This can easily result in missing emotional types and fail to meet the needs of diverse text scenarios.

[0006] Poor adaptability: Existing technologies do not achieve dynamic matching of text semantics, emotion, and audio. They can only generate speech based on fixed emotional audio, resulting in weak emotional and semantic correlation between synthesized speech and text context, which affects user experience.

[0007] Therefore, there is an urgent need for a TTS method that can generate multi-emotion target audio at low cost and automatically, and achieve accurate matching between the audio and the text semantics, in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0008] This invention provides an audio TTS method based on timbre conversion to construct multiple emotions and text semantic matching, in order to solve the aforementioned technical problems.

[0009] This invention provides an audio TTS method based on timbre conversion to construct multiple emotions and text semantic matching, comprising:

[0010] S1: Construct a multi-emotional prompt wav set for the target timbre:

[0011] S11: Collect original audio samples with multiple emotions, with the total duration of audio samples corresponding to each emotion being no less than n1 minutes;

[0012] S12: Collect initial training audio of the target timbre and build a target timbre conversion model;

[0013] S13: Determine if the total duration of the initial training audio is ≥ n2 minutes. If not, generate supplementary audio for the target timbre using the open-source speech model, merge it with the initial training audio to a total duration ≥ n2 minutes, and fine-tune the target timbre conversion model. If yes, directly fine-tune the target timbre conversion model to obtain the final target timbre conversion model.

[0014] S14: Input the original multi-emotion audio from S11 into the final target timbre conversion model to obtain the multi-emotion audio of the target timbre. Extract WAV format audio of a set time length from each emotion audio as prompt wav to form a multi-emotion prompt wav set.

[0015] S2: TTS speech generation based on text semantics and emotion matching:

[0016] S21: Get the text to be synthesized (input text);

[0017] S22: Use an emotion classification model to analyze the emotion category of the input text;

[0018] S23: Select candidate promptwavs that match the emotion category output in S22 from the multi-emotion prompt wav set. Calculate the semantic similarity between the input text and the corresponding text of the candidate prompt wavs based on the text embedding model. Select the candidate prompt wav with the highest semantic similarity as the final guiding audio.

[0019] S24: Input the input text, the final guiding audio, and the corresponding text into the TTS generative large model to generate the target speech audio.

[0020] Preferably, the emotion includes basic emotions and / or advanced emotions, and the audio fundamental frequency range covers the range of human speech voice;

[0021] The emotion classification model is a BERT-based text emotion classification model or an LLM.

[0022] Preferably, the basic emotions include: anger, sadness, excitement, surprise, happiness, fear, worry, frustration, and tension; the advanced emotions include contempt, anxiety, indifference, anger, sarcasm, and sincerity; the original audio for the multiple emotions uses the CASIA open-source audio dataset, and the dataset contains audio for 6 emotions, with each emotion audio lasting approximately 10 minutes in total.

[0023] Preferably, supplementary audio for the target timbre is generated using an open-source large-scale speech model, merged with the initial training audio to a total duration ≥ n² minutes, and the target timbre conversion model is fine-tuned, including:

[0024] S301: Collect target timbre audio manually as training samples;

[0025] S302: Check if the total duration of the training audio meets the requirement of 10 minutes. If not, supplement it by creating an open-source large speech model.

[0026] S303: Select a high-quality audio clip of about 10 seconds from S301 as the promptwav of the open-source speech model, which is used to guide the open-source speech model to generate speech with similar timbre.

[0027] S304: Captures speech audio with any timbre, with a total duration of more than 10 minutes;

[0028] S305: The arbitrary voice audio collected by S304 is converted into audio with the same voice as the target voice of promptwav selected by S303 through an open-source speech model, based on the target voice promptwav selected by S303, thus obtaining a large number of voice audios with the target voice.

[0029] S306: From the target voice speaking audio in S305, manually select high-quality target voice audio and supplement it with training audio based on S302 to bring the total duration of training audio to more than 10 minutes.

[0030] S307: Fine-tune the training target timbre model on a target timbre training set of more than 10 minutes;

[0031] The open-source speech models mentioned include SeedVC, StepAudio, or KimiAudio.

[0032] Preferably, the semantic similarity between the input text and the corresponding text of the candidate prompt wav is calculated based on the text embedding model, including:

[0033] The text embedding model is used to convert the input text and the corresponding text of the candidate prompt wav into embedding vectors respectively.

[0034] Calculate the cosine similarity between two embedding vectors, and use the cosine similarity value as the semantic similarity result.

[0035] Preferably, the TTS generative large model includes llasa-8B or HiggsaudioV2.

[0036] Preferably, after generating the target speech audio, the process further includes:

[0037] The text semantics are key-based extraction to obtain topic semantics, which are then concatenated with a random mask sequence to form the input sequence, resulting in the text sentiment representation vector S. text ;

[0038] Obtain the target spectrogram of the target speech audio, extract the individual pronunciation spectrograms for the same byte in the target spectrogram, and construct the combined spectrogram of the same letter, wherein the combined spectrogram contains at least one individual pronunciation spectrogram;

[0039] Based on the combined spectrum analysis of the timbre and pitch changes of the corresponding bytes, and combined with the audio position of the corresponding byte and the theoretical pronunciation rules based on the audio position, a pitch-timbre pair sequence is constructed.

[0040] The tone-timbre pairs of all phonological bytes are fused to obtain the emotion fluctuation vector Bx;

[0041] Extract the byte activity of each byte in the target graph, calculate the median Zw of all byte activities and the mean absolute deviation MA of the difference between each byte activity and the median, and obtain the sentiment confidence score.

[0042] The emotion fluctuation vector Bx is adjusted according to the emotion confidence level Ca to obtain the current fluctuation vector Dx;

[0043] The actual register feature vector Vs of the target speech audio is identified using a speech register recognition model, and combined with the text sentiment representation vector S. text The actual sentiment vector Sx is obtained by combining the current fluctuation vector Dx;

[0044] The coordination bias index Δ = γ1·Sim(Vs,Vs0) + γ2·Dist(Sx,Sx0) is calculated, where Vs0 is the original stylistic feature vector; Sx0 is the original sentiment vector; γ1 and γ2 are dynamic weights, satisfying γ1 + γ2 = 1. Each iteration follows the formula γ1·exp(0.05·Sim... txt-aud -0.5) Adaptive update, Sim txt-aud For S text Cosine similarity with Dx; Sim(Vs,Vs0) determines the similarity between Vs and Vs0; Dist(Sx,Sx0) determines the degree of difference between Sx and Sx0.

[0045] If Δ>τ, then Derivation of speech parameter adjustment amount Where τ is the set threshold; Here, represents the stylistic-emotion co-loss function; Grad() is the gradient of the loss function with respect to the input features; α is the parameter adjustment coefficient.

[0046] Will Input the TTS generative model and combine it with the original input text and the original filtered prompt wav to regenerate the target speech audio.

[0047] Preferably, a large number of speech audio recordings with the target timbre are obtained, including:

[0048] Extract the non-silent segments of the spoken audio with the arbitrary timbre, and perform a short-time Fourier transform on the non-silent segments to obtain the spectrum X(k,t). To achieve spectral reduction, where, The spectrum is the spectrum after spectral subtraction, and σ is the over-subtraction factor. To estimate the spectrum of noise, and simultaneously, according to To achieve wavelet denoising, where W j,l γ represents the wavelet coefficients; thresh() is the threshold function, and γ is the wavelet threshold. The spectrum after denoising;

[0049] Based on the text sequence output by speech recognition, and using semantic integrity scoring, sentence segments with integrity scores greater than the semantic integrity threshold are retained to obtain effective semantic segments;

[0050] High-quality audio is selected from the initial training audio of the target timbre, and the core timbre frequency band of the high-quality audio is enhanced based on a deep residual network to obtain the enhanced prompt wav;

[0051] Effective semantic segments and enhanced prompt wav are input into an open-source speech model, and supplementary audio of the target timbre is generated based on the dual-input attention fusion mechanism of the open-source speech model, serving as a large number of speech audio recordings of the target timbre.

[0052] Compared with the prior art, the beneficial effects of this application are as follows:

[0053] Reduce costs and achieve batch generation: By converting original audio with multiple emotions and timbre, the system can automatically generate multiple emotion prompt wavs of the target timbre in batches without the need for manual recording of the target timbre's multi-emotion audio, significantly reducing labor and time costs; at the same time, by supplementing the target timbre training audio with a large speech model, the problem of insufficient target timbre samples is solved, further reducing the difficulty of sample acquisition.

[0054] Comprehensive emotion coverage: Based on open-source datasets or custom-collected multi-emotion audio, it covers basic and advanced emotions, meeting the emotional needs of diverse text scenarios and avoiding the shortcomings of incomplete emotion coverage in existing technologies.

[0055] Improve TTS adaptability and quality: Through a dual screening mechanism of emotion classification and semantic similarity matching, ensure that the prompt wav is highly adapted to the emotion and semantics of the input text. Combined with the TTS large model to generate speech, the synthesized speech not only has the target timbre, but also accurately matches the text context, thus improving the user's auditory experience.

[0056] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0057] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0058] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0059] Figure 1 This is a flowchart of an audio TTS method based on timbre conversion to construct multiple emotions and text semantic matching in an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of the fine-tuning target timbre conversion model in an embodiment of the present invention. Detailed Implementation

[0061] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0062] This invention provides an audio TTS method based on timbre conversion to construct multiple emotions and match text semantics, such as... Figure 1 As shown, it includes:

[0063] S1: Construct a multi-emotional prompt wav set for the target timbre:

[0064] S11: Collect original audio samples with multiple emotions, with the total duration of audio samples corresponding to each emotion being no less than n1 minutes;

[0065] S12: Collect initial training audio of the target timbre and build a target timbre conversion model;

[0066] S13: Determine if the total duration of the initial training audio is ≥ n2 minutes. If not, generate supplementary audio for the target timbre using the open-source speech model, merge it with the initial training audio to a total duration ≥ n2 minutes, and fine-tune the target timbre conversion model. If yes, directly fine-tune the target timbre conversion model to obtain the final target timbre conversion model.

[0067] S14: Input the original multi-emotion audio from S11 into the final target timbre conversion model to obtain the multi-emotion audio of the target timbre. Extract WAV format audio of a set time length from each emotion audio as prompt wav to form a multi-emotion prompt wav set.

[0068] S2: TTS speech generation based on text semantics and emotion matching:

[0069] S21: Get the text to be synthesized (input text);

[0070] S22: Use an emotion classification model to analyze the emotion category of the input text;

[0071] S23: Select candidate promptwavs that match the emotion category output in S22 from the multi-emotion prompt wav set. Calculate the semantic similarity between the input text and the corresponding text of the candidate prompt wavs based on the text embedding model. Select the candidate prompt wav with the highest semantic similarity as the final guiding audio.

[0072] S24: Input the input text, the final guiding audio, and the corresponding text into the TTS generative large model to generate the target speech audio.

[0073] Preferably, the emotion includes basic emotions and / or advanced emotions, and the audio fundamental frequency range covers the range of human speech voice;

[0074] The emotion classification model is a BERT-based text emotion classification model or an LLM.

[0075] Preferably, the basic emotions include: anger, sadness, excitement, surprise, happiness, fear, worry, frustration, and tension; the advanced emotions include contempt, anxiety, indifference, anger, sarcasm, and sincerity; the original audio for the multiple emotions uses the CASIA open-source audio dataset, and the dataset contains audio for 6 emotions, with each emotion audio lasting approximately 10 minutes in total.

[0076] Preferably, supplementary audio for the target timbre is generated using an open-source large-scale speech model, merged with the initial training audio to a total duration ≥ n² minutes, and the target timbre conversion model is fine-tuned, such as... Figure 2 As shown, it includes:

[0077] S301: Collect target timbre audio manually as training samples;

[0078] S302: Check if the total duration of the training audio meets the requirement of 10 minutes. If not, supplement it by creating an open-source large speech model.

[0079] S303: Select a high-quality audio clip of about 10 seconds from S301 as the promptwav of the open-source speech model, which is used to guide the open-source speech model to generate speech with similar timbre.

[0080] S304: Captures speech audio with any timbre, with a total duration of more than 10 minutes;

[0081] S305: The arbitrary voice audio collected by S304 is converted into audio with the same voice as the target voice of promptwav selected by S303 through an open-source speech model, based on the target voice promptwav selected by S303, thus obtaining a large number of voice audios with the target voice.

[0082] S306: From the target voice speaking audio in S305, manually select high-quality target voice audio and supplement it with training audio based on S302 to bring the total duration of training audio to more than 10 minutes.

[0083] S307: Fine-tune the training target timbre model on a target timbre training set of more than 10 minutes;

[0084] The open-source speech models mentioned include SeedVC, StepAudio, or KimiAudio.

[0085] Preferably, the semantic similarity between the input text and the corresponding text of the candidate prompt wav is calculated based on the text embedding model, including:

[0086] The text embedding model is used to convert the input text and the corresponding text of the candidate prompt wav into embedding vectors respectively.

[0087] Calculate the cosine similarity between two embedding vectors, and use the cosine similarity value as the semantic similarity result.

[0088] Preferably, the TTS generative large model includes llasa-8B or HiggsaudioV2.

[0089] In this embodiment, the CASIA open-source dataset was used to extract audio recordings of six emotions: surprise, sadness, neutral, happy, fear, and angry. Each emotion's audio recording was 10 minutes long, with a sampling rate of 16kHz and mono. A neutral emotion audio recording of a male speaker was manually recorded, initially 8 minutes long (less than 10 minutes), to train the target voice conversion model (built using Retrieval-based Voice Conversion technology). If the initial training audio duration is determined to be 8 minutes < 10 minutes, execute steps S132-S136. Select a 10-second audio segment with noise-free sound and a stable speaking rate from the 8-minute initial audio as the prompt wav for the SeedVC model. Collect audio recordings of a female speaker with arbitrary timbre, totaling 15 minutes in duration (16kHz sampling rate, mono). Input the 15-minute female audio and the 10-second prompt wav into the SeedVC model to generate supplementary audio for a neutral male voice, totaling 12 minutes in duration. Manually screen the supplementary audio, removing 3 minutes of poor-quality segments and retaining 9 minutes of high-quality supplementary audio. Merge this with the initial 8-minute audio, resulting in a total duration of 17 minutes (≥ 10 minutes). Based on the 17-minute merged audio, fine-tune the Retrieval based Voice Conversion model to obtain the neutral male voice target timbre conversion model.

[0090] In this embodiment, the six emotional audio samples from S11 are input into the neutral male voice conversion model to obtain the six emotional neutral male voice audio samples; a 10-second high-quality segment is extracted from each emotional audio sample (e.g., the 10-second audio sample of "The weather is so nice today" for the "happy" emotion) to form a set of prompt wav corresponding to the six emotions.

[0091] For example, to generate a TTS message wishing you a Happy New Year and all the best, the specific steps are as follows:

[0092] Input text to be synthesized: Happy New Year, wishing you all the best. Use an LLM (e.g., GPT-3.5) to execute the following instructions: Based on the input text, select the most suitable emotion from surprise, sad, neutral, happy, fear, and angry; Input text: Happy New Year, wishing you all the best, LLM outputs the emotion category as happy. In the promptwav set of S14, select the candidate prompt wav corresponding to the happy emotion (its prompt text is "The weather is nice today"); use the m3e model to convert the input text and prompt text into embedding vectors, calculate the cosine similarity to 0.82 (high similarity), and determine this prompt wav as the final guiding audio. Input the input text, the 10-second happy emotion prompt wav, and the prompt text "The weather is nice today" into the llasa-8BTTS model to generate a neutral male voice with the happy emotion, approximately 5 seconds long, with a sampling rate of 16kHz, mono.

[0093] In this embodiment, the synthesized speech not only conforms to the target neutral male voice timbre, but also matches the text context of "Happy New Year" with a happy emotion. It has strong semantic relevance, clear sound quality, and no obvious synthesis traces, thus verifying the effectiveness of the method of the present invention.

[0094] The beneficial effects of the above technical solution are: reduced costs and batch generation: by converting original audio with multiple emotions and timbre, the multi-emotion audio of the target timbre can be automatically generated in batches without the need for manual recording of multi-emotion audio, which significantly reduces labor and time costs; at the same time, by supplementing the training audio of the target timbre with a large speech model, the problem of insufficient target timbre samples is solved, further reducing the difficulty of obtaining samples.

[0095] Comprehensive emotion coverage: Based on open-source datasets or custom-collected multi-emotion audio, it covers basic and advanced emotions, meeting the emotional needs of diverse text scenarios and avoiding the shortcomings of incomplete emotion coverage in existing technologies.

[0096] Improve TTS adaptability and quality: Through a dual screening mechanism of emotion classification and semantic similarity matching, ensure that the prompt wav is highly adapted to the emotion and semantics of the input text. Combined with the TTS large model to generate speech, the synthesized speech not only has the target timbre, but also accurately matches the text context, thus improving the user's auditory experience.

[0097] This invention provides an audio TTS method based on timbre conversion to construct multi-emotion and text semantic matching, which, after generating the target speech audio, further includes:

[0098] The text semantics are key-based extraction to obtain topic semantics, which are then concatenated with a random mask sequence to form the input sequence, resulting in the text sentiment representation vector S. text ;

[0099] Obtain the target spectrogram of the target speech audio, extract the individual pronunciation spectrograms for the same byte in the target spectrogram, and construct the combined spectrogram of the same letter, wherein the combined spectrogram contains at least one individual pronunciation spectrogram;

[0100] Based on the combined spectrum analysis of the timbre and pitch changes of the corresponding bytes, and combined with the audio position of the corresponding byte and the theoretical pronunciation rules based on the audio position, a pitch-timbre pair sequence is constructed.

[0101] The tone-timbre pairs of all phonological bytes are fused to obtain the emotion fluctuation vector Bx;

[0102] Extract the byte activity of each byte in the target graph, calculate the median Zw of all byte activities and the mean absolute deviation MA of the difference between each byte activity and the median, and obtain the sentiment confidence score.

[0103] The emotion fluctuation vector Bx is adjusted according to the emotion confidence level Ca to obtain the current fluctuation vector Dx;

[0104] The actual register feature vector Vs of the target speech audio is identified using a speech register recognition model, and combined with the text sentiment representation vector S. text The actual sentiment vector Sx is obtained by combining the current fluctuation vector Dx;

[0105] The coordination bias index Δ = γ1·Sim(Vs,Vs0) + γ2·Dist(Sx,Sx0) is calculated, where Vs0 is the original stylistic feature vector; Sx0 is the original sentiment vector; γ1 and γ2 are dynamic weights, satisfying γ1 + γ2 = 1. Each iteration follows the formula γ1·exp(0.05·Sim... txt-aud -0.5) Adaptive update, Sim txt-aud For S text Cosine similarity with Dx; Sim(Vs,Vs0) determines the similarity between Vs and Vs0; Dist(Sx,Sx0) determines the degree of difference between Sx and Sx0.

[0106] If Δ>τ, then Derivation of speech parameter adjustment amount Where τ is the set threshold; α is the register-emotion co-loss function; Grad() is the gradient of the loss function with respect to the input features; α is the parameter adjustment coefficient, which is optimized through validation set experiments. A value of 0.1 yields the best speech optimization effect.

[0107] In this embodiment, the initial values of γ1 and γ2 are 0.5 respectively for initial balancing, and subsequent updates of γ1 and γ2 are realized based on the iterative formula.

[0108] In this embodiment, The mathematical meaning of the gradient of Grad() is the direction in which the loss function rises fastest. Taking the negative gradient is the direction in which it drops fastest, that is, the optimization direction.

[0109] Put Input into the TTS generative large model, and combine the original input text and the originally selected prompt wav to regenerate the target speech audio.

[0110] In this embodiment, text semantics refers to the core meaning and content information expressed by the text. For example, in the sentence "In spring, the flowers are in full bloom and I go to enjoy the flowers with great joy", its semantics is to describe the event of the flowers blooming in spring and oneself going to enjoy the flowers with joy.

[0111] Key extraction is to extract important keywords or key sentences from the text by means of natural language processing technologies such as the TF-IDF algorithm and the TextRank algorithm. Taking "In spring, the flowers are in full bloom and I go to enjoy the flowers with great joy" as an example, key information such as spring, flowers in full bloom, joy, and enjoy the flowers is extracted.

[0112] Thematic semantics is the content that can represent the core theme of the text obtained by key extraction. For example, from the above text, the happy mood when enjoying the spring flowers can be extracted as the thematic semantics.

[0113] A random masked sequence is a sequence in natural language processing where part of the text content is randomly replaced with special tokens (such as [MASK]) to enhance the model's learning of semantics. For example, take "In spring, [MASK] flowers are in full bloom and I go to enjoy the flowers with great [MASK]" as a random masked sequence, and then splice it with the thematic semantics of the happy mood when enjoying the spring flowers to form an input sequence.

[0114] The text emotion representation vector is a vector obtained by converting the spliced input sequence through the embedding layer of a pre-trained language model (such as BERT), which is used to represent the emotion characteristics of the text. The happy emotion in the above text will be numerically captured by this vector.

[0115] The target spectrogram is a spectrogram obtained by performing signal processing on the target speech audio, which shows the energy distribution of the audio at different times and frequencies. The target spectrogram is obtained by performing a short-time Fourier transform on the audio of "Happy to enjoy the spring flowers" using the librosa library in Python.

[0116] A byte is the text character corresponding to the smallest pronunciation unit in speech, such as the character "春" corresponding to its pronunciation.

[0117] A single pronunciation spectrum is the spectrum corresponding to the pronunciation period of a single byte extracted from the target spectrum for the pronunciation of a single byte. For example, the spectrum segment corresponding to the pronunciation of the character "春" (assuming it is in the audio period of 0.5 - 1 second) is the single pronunciation spectrum.

[0118] A combined spectrum is a spectrum formed by combining at least one single pronunciation spectrum of the same letter (or the same pronunciation unit). For example, the character "春" has 3 pronunciation spectra at different speech speeds, and they are combined into a combined spectrum.

[0119] Timbre change is the change of timbre during the pronunciation of a byte. For example, for the same character "呀", when pronounced with happy and sad emotions, there will be differences in timbre, which is judged by analyzing features such as the harmonic structure of the spectrum in the combined spectrum.

[0120] Pitch change is the change determined by the frequency pitch of the byte pronunciation. For example, in the word "快乐", the pitch of the character "乐" decreases from the beginning to the end, and the pitch change can be analyzed through the change trend of the frequency axis in the combined spectrum.

[0121] Audio position is the time position of the byte pronunciation in the entire target speech audio. For example, the character "春" is pronounced in the audio from 0.5 - 1 second.

[0122] The theoretical pronunciation rule is the norm of byte pronunciation in a language. For example, the standard pronunciation (such as the pronunciation duration and frequency range of the initial consonant and final vowel) of the character "春" in Chinese can be obtained from phonetic materials. It should be noted that different characters may have different pronunciations in different combinations. For example, a character may change from the first tone to the neutral tone.

[0123] The pitch - timbre pair sequence is a sequence formed by corresponding the pitch change and timbre change of each byte. For example, each byte corresponds to a pair of (pitch feature, timbre feature), and they are arranged in a sequence in the order of 春, 日, 赏, 花.

[0124] The fusion process is to integrate all the pitch - timbre pair sequences of bytes into a vector through methods such as vector splicing and weighted summation. For example, after using the attention mechanism to assign weights to the pairs of different bytes such as "春" and "日" and then summing them, the emotional fluctuation vector Bx is obtained.

[0125] The emotional fluctuation vector Bx is a vector that can reflect the emotional fluctuation situation with the change of bytes in the speech after fusion. For example, in the speech with a happy emotion, the pitch - timbre changes of the bytes are more active, and the numerical change of Bx is also more active.

[0126] In this embodiment, taking the target speech audio of the text "今天真开心!" as an example, the process of obtaining the emotional fluctuation vector Bx is as follows:

[0127] The target speech audio is split into bytes such as "今", "天", "真", "开", "心", "!" according to pronunciation, and the pronunciation periods of each byte are subjected to spectral analysis through short-time Fourier transform (STFT) to obtain a single pronunciation spectrum. For example:

[0128] "今" corresponds to the audio period of 0 - 0.3 seconds, and through STFT, "single pronunciation spectrum 1" is obtained;

[0129] "开" corresponds to the audio period of 0.9 - 1.3 seconds, which contains two sub-periods of 0.9 - 1.1 seconds and 1.1 - 1.3 seconds. Two single pronunciation spectra are extracted and combined into a combined spectrum.

[0130] The pitch change is reflected by the fundamental frequency, and the timbre change is reflected by the Mel-frequency cepstral coefficients (MFCC) (simplified to take the first 3 dimensions). A (pitch feature, timbre feature) pair is constructed for each byte:

[0131] "今": fundamental frequency 180 Hz (pitch feature

[180] ), MFCC is [0.2, 0.1, 0.3] (timbre feature) → pair: (

[180] , [0.2, 0.1, 0.3]);

[0132] "开": fundamental frequency 200 Hz in the sub-period of 0.9 - 1.1 seconds, fundamental frequency 220 Hz in the sub-period of 1.1 - 1.3 seconds (pitch feature [200, 220]); the MFCC of the sub-periods are [0.4, 0.3, 0.4] and [0.5, 0.4, 0.5] respectively, and are spliced into [0.4, 0.3, 0.4, 0.5, 0.4, 0.5] (timbre feature) → pair: ([200, 220], [0.4, 0.3, 0.4, 0.5, 0.4, 0.5]);

[0133] Similarly, for "天", "真", "心", "!", the corresponding "pitch-timbre pairs" are also obtained, and finally, a pitch-timbre pair sequence is formed in the pronunciation order: [the pair of "今", the pair of "天", the pair of "真", the pair of "开", the pair of "心", the pair of "!"].

[0134] The attention mechanism is used to perform weighted fusion on the "pairs" of each byte in the sequence (higher weights are assigned to the key emotion bytes "开" and "心"):

[0135] Set the attention weights: 0.1 for "今", 0.1 for "天", 0.1 for "真", 0.2 for "开", 0.3 for "心", 0.2 for "!";

[0136] The feature vectors of each "pitch-timbre pair" are spliced (for example,

[180] of "今" and [0.2, 0.1, 0.3] are spliced into [180, 0.2, 0.1, 0.3]);

[0137] After multiplying each splicing vector by the corresponding weight and summing them up, the final emotional fluctuation vector Bx is obtained.

[0138] For example, after weighted summation of each vector, some dimensions of Bx are [204, 44.28, 0.28, 0.34, …], and the vector captures the fluctuation trend of the rising fundamental frequency and brighter timbre under the "happy" emotion.

[0139] In this way, the emotional fluctuation vector Bx integrates the pitch and timbre change rules of each byte in the speech, and can accurately reflect the dynamic fluctuation characteristics of emotions such as "happy" in the speech.

[0140] Byte activity is an index to measure the activity degree of a byte during pronunciation, such as energy change and frequency change in the target spectrum. It is represented by calculating the energy variance of the spectrum corresponding to the byte, and a larger variance indicates a higher activity.

[0141] The median Zw is the median of the activities of all bytes, which is obtained by sorting the activities of all bytes and taking the middle value. For example, if the activities of 5 bytes are [0.3, 0.5, 0.6, 0.7, 0.9], the median Zw is 0.6.

[0142] The mean absolute deviation MA is the average of the absolute values of the differences between the activities of each byte and the median Zw.

[0143] In this embodiment, the current fluctuation vector Dx is a vector obtained by adjusting the emotional fluctuation vector Bx through scaling or weighting with the emotional confidence Ca. For example, if Ca = 0.7, each element of Bx is multiplied by 0.7 to obtain Dx, making the vector more conform to the confidence level of the current emotion.

[0144] The speech style recognition model is a model used to identify which style the speech belongs to (such as formal, spoken, sad-tone style, etc.). It is trained by a deep neural network (such as CNN + LSTM) with more than 1000 labeled speech data of styles. When inputting the audio of "It's really happy to enjoy flowers in spring", it outputs a style feature vector.

[0145] The actual style feature vector Vs is a vector output by the speech style recognition model, representing the actual style of the target speech. For example, if the speech of "It's really happy to enjoy flowers in spring" is recognized as "a pleasant spoken style", the corresponding vector is Vs.

[0146] The actual emotion vector Sx is a vector obtained by fusing the text emotion representation vector and the current fluctuation vector Dx, and is used to comprehensively reflect the emotion characteristics of the text and speech.

[0147] The co-bias index Δ is an index to measure the degree of co-bias between the style and emotion. The larger Δ is, the greater the deviation.

[0148] The original register feature vector Vs0 is the feature vector corresponding to the desired register of the target speech audio. For example, if we want the speech of "Spring flowers are so enjoyable" to be in a "formal and pleasant register", the corresponding vector is Vs0.

[0149] The original emotion vector Sx0 is the vector corresponding to the expected emotion of the target speech audio. For example, if we want to express the emotion of "joy", the corresponding vector is Sx0.

[0150] Dynamic weights γ1 and γ2 are based on Sim txt-aud Dynamic adjustment, Sim txt-aud When the value is high, γ1 increases, indicating a greater emphasis on stylistic similarity.

[0151] In this embodiment, the values ​​of Sim(Vs,Vs0) and Dist(Sx,Sx0) range from 0 to 1.

[0152] The threshold is a pre-set threshold for determining whether adjustment is needed; based on the experiment, it was set to 0.3.

[0153] Grad() is the gradient of the loss function with respect to the input features, that is, the gradient obtained by taking the derivative of the loss function with respect to the input features, which is used to guide the direction of speech parameter adjustment.

[0154] Large-scale TTS generative models are those that can convert text into speech (such as llasa-8B, HiggsaudioV2, etc.), with input text, speech prompt information (prompt wav), and parameter adjustments. Generate optimized speech.

[0155] The original input text is the original text that needs to be synthesized into speech, such as "Enjoying flowers in spring is so much fun".

[0156] The original selected prompt wavs were guided audio clips that matched the text's emotion and semantics, selected from a multi-emotion prompt wav set. For example, the audio clip "The weather is so nice today" matched the emotion of "Enjoying flowers in spring is so enjoyable".

[0157] The beneficial effects of the above technical solution are: by analyzing and integrating multiple dimensions such as text semantics, speech spectrum, emotion and stylistic features, and dynamically adjusting speech parameters, the generated target speech audio is more in line with the text semantics in terms of emotional expression and stylistic style, while improving the coordination and consistency between emotion and stylistic style, and ultimately generating more natural and context-appropriate high-quality speech.

[0158] This invention provides an audio TTS method based on timbre conversion to construct multi-emotion and text semantic matching, obtaining a large number of speaking audio recordings with target timbres, including:

[0159] Extract the non-silent segments of the spoken audio with the arbitrary timbre, and perform a short-time Fourier transform on the non-silent segments to obtain the spectrum X(k,t). To achieve spectral reduction, where, The spectrum is the spectrum after spectral subtraction, and σ is the over-subtraction factor. To estimate the spectrum of noise, and simultaneously, according to To achieve wavelet denoising, where W j,l γ represents the wavelet coefficients; thresh() is the threshold function, and γ is the wavelet threshold. The spectrum after denoising;

[0160] Based on the text sequence output by speech recognition, and using semantic integrity scoring, sentence segments with integrity scores greater than the semantic integrity threshold are retained to obtain effective semantic segments;

[0161] High-quality audio is selected from the initial training audio of the target timbre, and the core timbre frequency band of the high-quality audio is enhanced based on a deep residual network to obtain the enhanced prompt wav;

[0162] Effective semantic segments and enhanced prompt wav are input into an open-source speech model, and supplementary audio of the target timbre is generated based on the dual-input attention fusion mechanism of the open-source speech model, serving as a large number of speech audio recordings of the target timbre.

[0163] In this embodiment, "arbitrary timbre" refers to speech audio with arbitrary vocal characteristics, such as a typical woman's everyday speech audio.

[0164] Non-silent segments are the parts of the audio that contain valid speech (not completely silent). They are extracted using an energy threshold detection method. The energy of the audio frame is calculated, and when the energy exceeds a threshold calculated based on background noise, it is determined to be a non-silent segment.

[0165] The Short Time Fourier Transform (STFT) divides a non-silent segment of a signal in the time domain into multiple short time frames, performs a Fourier transform on each frame, and obtains the spectrum that changes over time (reflecting the frequency components at different times).

[0166] In this embodiment, the over-subtraction factor is a parameter that adjusts the degree of noise reduction, and is set to 1.2. It needs to be optimized according to the actual noise situation. An over-subtraction factor that is too large can easily lead to speech distortion, while one that is too small will result in insufficient noise reduction. It is usually selected in the range of 1.0 to 1.5 through experiments.

[0167] In this embodiment, the PyWavelets library is used, the db4 wavelet is selected, the decomposition level is 5, and a soft thresholding function is applied to the audio coefficients. When |W j,l When |≥γ, thresh(|W j,l |,γ)=|W j,l|-γ, otherwise 0, where γ is determined by multiplying the standard deviation of the wavelet coefficients by 2lgN, where N is the signal length.

[0168] wavelet coefficients W j,l The coefficients obtained after time wavelet transform represent the components of the signal at different frequencies and time scales. Large coefficients correspond to the core features of the signal, while small coefficients often contain noise.

[0169] Speech recognition is a technology that converts audio into a text sequence. This paper uses an open-source model to process the denoised audio and converts the speech "The weather is nice today" into the text sequence "The weather is nice today".

[0170] A text sequence is a combination of text output by speech recognition; it is a sequence of continuous sentences or words.

[0171] Semantic integrity scoring is a quantitative score that measures whether a text fragment possesses complete semantic meaning. It is achieved by calculating perplexity using a language model (such as BERT) or by scoring using a pre-trained semantic integrity model. For example, the text "Today I went to the park" is semantically complete and receives a high score, while "Today I went to...park" (incomplete) receives a low score.

[0172] The semantic completeness threshold is a set score standard; text fragments scoring above this value are considered semantically complete. In the experiment, the semantic completeness threshold was set to 0.7, and fragments with scores ≥0.7 were retained.

[0173] Valid semantic segments are text segments that are semantically complete after being filtered by semantic integrity scoring. They serve as semantic templates for generating target timbre audio, such as the filtered phrase "The flowers in spring are blooming beautifully".

[0174] The initial training audio for the target timbre is the original audio used to train the target timbre conversion model. For example, it is the neutral speech audio of a male announcer.

[0175] High-quality audio clips are selected from the initial training audio of the target timbre. They must meet conditions such as no obvious noise, stable speech rate, and clear timbre. They are selected through manual listening or automatic detection (such as signal-to-noise ratio detection and fundamental frequency stability analysis). For example, a 10-second male speech clip with no noise and a full timbre can be selected from the initial audio.

[0176] Deep Residual Networks (ResNet) are deep learning networks with residual connections, enabling effective feature extraction and enhancement. This design utilizes a ResNet-based audio spectrum enhancement network to enhance the core timbre frequency bands of high-quality audio. It takes the spectrum of high-quality audio as input and enhances the features of the core timbre frequency bands (such as the fundamental frequency and harmonic bands that determine the characteristics of the human voice, determined by analyzing the energy concentration region of the target timbre spectrum) through residual blocks, outputting the enhanced spectrum.

[0177] The core timbre frequency band is the key frequency range that determines the uniqueness of a timbre. Different timbres (such as male and female timbres) have different core frequency bands, which are determined by analyzing the spectral energy distribution of the target timbre. For example, the core frequency band of male timbres is mostly concentrated around 80-1000Hz.

[0178] The enhanced prompt wav is a high-quality audio file that has been enhanced in the core timbre frequency bands using a deep residual network. Its target timbre features are more prominent, and it is used to guide open-source speech models to generate new audio that fits the target timbre. For example, the enhanced male speech segment can more clearly convey the characteristics of the male timbre.

[0179] In this embodiment, the dual-input attention fusion mechanism is an attention mechanism in the open-source speech model that simultaneously processes effective semantic segments (text) and reinforcement prompt wav (audio). The model is divided into a text branch and an audio branch, which extract text semantic features and audio timbre features respectively, and then fuse the two information through an attention layer so that the generated audio matches both the text semantics and the timbre of the prompt audio.

[0180] Target timbre supplementary audio is the final generated new audio with the target timbre, used to expand the training data of the target timbre conversion model. For example, given the valid semantic fragment "The sun is shining brightly today" and the enhanced prompt wav (male timbre), it generates the audio "The sun is shining brightly today" in a male timbre.

[0181] The beneficial effects of the above technical solution are: improving audio purity through spectral subtraction and wavelet denoising, ensuring text content quality through semantic integrity scoring, strengthening the core timbre features of the guiding audio through deep residual networks, and efficiently generating supplementary audio that matches the target timbre and is semantically complete by relying on the dual-input attention fusion mechanism of the open-source speech model. This provides high-quality and sufficient training data for subsequent timbre conversion and TTS generation, and improves the timbre consistency and naturalness of speech generation.

[0182] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An audio TTS method based on timbre conversion to construct multi-emotion and text semantic matching, characterized in that, include: S1: Construct a multi-emotional prompt wav set for the target timbre: S11: Collect original audio samples with multiple emotions, with the total duration of audio samples corresponding to each emotion being no less than n1 minutes; S12: Collect initial training audio of the target timbre and build a target timbre conversion model; S13: Determine if the total duration of the initial training audio is ≥ n2 minutes. If not, generate supplementary audio for the target timbre using the open-source speech model, merge it with the initial training audio to a total duration ≥ n2 minutes, and fine-tune the target timbre conversion model. If yes, directly fine-tune the target timbre conversion model to obtain the final target timbre conversion model. S14: Input the original multi-emotion audio from S11 into the final target timbre conversion model to obtain the multi-emotion audio of the target timbre. Extract WAV format audio of a set time length from each emotion audio as prompt wav to form a multi-emotion prompt wav set. S2: TTS speech generation based on text semantics and emotion matching: S21: Get the text to be synthesized (input text); S22: Use an emotion classification model to analyze the emotion category of the input text; S23: Select candidate prompt wavs that match the emotion category output in S22 from the multi-emotion prompt wav set. Calculate the semantic similarity between the input text and the corresponding text of the candidate prompt wavs based on the text embedding model. Select the candidate prompt wav with the highest semantic similarity as the final guiding audio. S24: Input the input text, the final guiding audio, and the corresponding text into the TTS generative large model to generate the target speech audio.

2. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, The emotions include basic emotions and / or advanced emotions, and the audio fundamental frequency range covers the range of human speech voice. The emotion classification model is a BERT-based text emotion classification model or an LLM.

3. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, The basic emotions include: anger, sadness, excitement, surprise, happiness, fear, worry, frustration, and tension. The advanced emotions include contempt, anxiety, indifference, anger, sarcasm, and sincerity. The original audio for the multiple emotions uses the CASIA open-source audio dataset, which contains audio for 6 different emotions, with each emotion audio lasting approximately 10 minutes.

4. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, The target timbre supplementary audio is generated using an open-source large speech model, merged with the initial training audio to a total duration ≥ n² minutes, and the target timbre conversion model is fine-tuned, including: S301: Collect target timbre audio manually as training samples; S302: Check if the total duration of the training audio meets the requirement of 10 minutes. If not, supplement it by creating an open-source large speech model. S303: Select a high-quality audio clip of about 10 seconds from S301 as the prompt wav of the open-source speech model, which is used to guide the open-source speech model to generate speech with similar timbre. S304: Captures speech audio with any timbre, with a total duration of more than 10 minutes; S305: Take the speech audio with arbitrary timbre collected by S304, and based on the prompt wav of the target timbre selected by S303, use the open-source speech model to convert the speech audio with arbitrary timbre of S304 into audio with the same timbre as the target timbre of the prompt wav selected by S303, and obtain a large number of speech audio with the target timbre. S306: From the target voice speaking audio in S305, manually select high-quality target voice audio and supplement it with training audio based on S302 to bring the total duration of training audio to more than 10 minutes. S307: Fine-tune the training target timbre model on a target timbre training set of more than 10 minutes; The open-source speech models mentioned include SeedVC, StepAudio, or KimiAudio.

5. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, The semantic similarity between the input text and the corresponding text of the candidate prompt wav is calculated based on the text embedding model, including: The text embedding model is used to convert the input text and the corresponding text of the candidate prompt wav into embedding vectors respectively. Calculate the cosine similarity between two embedding vectors, and use the cosine similarity value as the semantic similarity result.

6. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, The TTS generative large model includes llasa-8B or HiggsaudioV2.

7. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 1, characterized in that, After generating the target speech audio, the process also includes: The text semantics are key-based extraction to obtain topic semantics, which are then concatenated with a random mask sequence to form the input sequence, resulting in the text sentiment representation vector S. text ; Obtain the target spectrogram of the target speech audio, extract the individual pronunciation spectrograms for the same byte in the target spectrogram, and construct the combined spectrogram of the same letter, wherein the combined spectrogram contains at least one individual pronunciation spectrogram; Based on the combined spectrum analysis of the timbre and pitch changes of the corresponding bytes, and combined with the audio position of the corresponding byte and the theoretical pronunciation rules based on the audio position, a pitch-timbre pair sequence is constructed. The tone-timbre pairs of all phonological bytes are fused to obtain the emotion fluctuation vector Bx; Extract the byte activity of each byte in the target graph, calculate the median Zw of all byte activities and the mean absolute deviation MA of the difference between each byte activity and the median, and obtain the sentiment confidence score. The emotion fluctuation vector Bx is adjusted according to the emotion confidence level Ca to obtain the current fluctuation vector Dx; The actual register feature vector Vs of the target speech audio is identified using a speech register recognition model, and combined with the text sentiment representation vector S. text The actual sentiment vector Sx is obtained by combining the current fluctuation vector Dx; The coordination bias index Δ = γ1·Sim(Vs,Vs0) + γ2·Dist(Sx,Sx0) is calculated, where Vs0 is the original stylistic feature vector; Sx0 is the original sentiment vector; γ1 and γ2 are dynamic weights, satisfying γ1 + γ2 = 1. Each iteration follows the formula γ1·exp(0.05·Sim... txt-aud -0.5) Adaptive update, Sim txt-aud For S text Cosine similarity with Dx; Sim(Vs,Vs0) determines the similarity between Vs and Vs0; Dist(Sx,Sx0) determines the degree of difference between Sx and Sx0. If Δ>τ, then Derivation of speech parameter adjustment amount Where τ is the set threshold; Here, represents the stylistic-emotion co-loss function; Grad() is the gradient of the loss function with respect to the input features; α is the parameter adjustment coefficient. Will Input the TTS generative model and combine it with the original input text and the original filtered prompt wav to regenerate the target speech audio.

8. The audio TTS method based on timbre conversion for constructing multi-emotion and text semantic matching according to claim 4, characterized in that, Obtain a large number of speech audio recordings of the target timbre, including: Extract the non-silent segments of the spoken audio with the arbitrary timbre, and perform a short-time Fourier transform on the non-silent segments to obtain the spectrum X(k,t). To achieve spectral reduction, where, The spectrum is the spectrum after spectral subtraction, and σ is the over-subtraction factor. To estimate the spectrum of noise, and simultaneously, according to To achieve wavelet denoising, where W j,l γ represents the wavelet coefficients; thresh() is the threshold function, and γ is the wavelet threshold. The spectrum after denoising; Based on the text sequence output by speech recognition, and using semantic integrity scoring, sentence segments with integrity scores greater than the semantic integrity threshold are retained to obtain effective semantic segments; High-quality audio is selected from the initial training audio of the target timbre, and the core timbre frequency band of the high-quality audio is enhanced based on a deep residual network to obtain the enhanced prompt wav; Effective semantic segments and enhanced prompt wav are input into an open-source speech model, and supplementary audio of the target timbre is generated based on the dual-input attention fusion mechanism of the open-source speech model, serving as a large number of speech audio recordings of the target timbre.

Images