Speech synthesis method and device, equipment and storage medium

Abstract

The invention relates to the technical field of speech synthesis, and discloses a speech synthesis method and device, equipment and a storage medium. The speech synthesis method comprises the following steps: acquiring to-be-processed text data and speech data; converting the text data into a text word segmentation sequence, and extracting speaker embedding features and emotion embedding features of the voice data; inputting the text word segmentation sequence, the speaker embedded features and the emotion embedded features into a voice semantic synthesis model for processing, and outputting a voice semantic word segmentation sequence; and converting the speech semantic word segmentation sequence into a Mel spectrum frame sequence, performing speech waveform synthesis on the Mel spectrum frame sequence, and outputting synthesized speech. According to the invention, the emotion expressive force and controllability of the synthesized speech are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a speech synthesis method, device, equipment and storage medium. BACKGROUND

[0002] As an important branch of artificial intelligence, speech synthesis technology (Text-to-Speech, TTS) has gradually shown its application potential with the rapid development of large models in recent years. Speech synthesis technology has evolved from a simple text-to-speech tool to a complex system capable of generating natural and expressive speech, and is playing an increasingly important role in modern technology society.

[0003] Current speech synthesis methods based on large models on the market still have significant defects. Most of these methods use emotion data collected from the Internet as training data, but such emotion data has many problems. First, the emotion distribution is uneven, and emotion data on the Internet is often biased towards some mainstream emotions, while data of other emotion types is relatively scarce, resulting in inconsistent performance of the model when processing different emotions. Second, the expression of different emotion strengths is not stable, and due to the lack of standardized emotion intensity labeling, the model cannot accurately grasp and express the subtle differences of emotions, making the emotional expression of synthesized speech not natural and accurate. Most importantly, existing technologies lack consideration of emotion controllability, and users cannot accurately control the emotion type and intensity in synthesized speech, thereby limiting the application of speech synthesis in scenarios that require accurate emotional expression.

[0004] Therefore, how to solve the problems of emotion data imbalance, unstable expression of emotion strength, and lack of emotion controllability has become a key technical problem to be solved in the development of current speech synthesis technology based on large models. Developing a speech synthesis method that can effectively process emotion data and achieve accurate emotion control is of great significance for improving the practicality and user experience of speech synthesis technology. SUMMARY

[0005] The main purpose of the present application is to provide a speech synthesis method, device, equipment and storage medium, which aims to solve the technical problems of emotion data imbalance, unstable expression of emotion strength, and lack of emotion controllability in existing speech synthesis technology.

[0006] The first aspect of the present application provides a speech synthesis method, which comprises: obtaining text data and speech data to be processed; converting the text data into a text segmentation sequence, and extracting speaker embedding features and emotion embedding features of the speech data; input the text word sequence, the speaker embedding feature and the emotion embedding feature into a speech semantic synthesis model for processing, and output a speech semantic word sequence; convert the speech semantic word sequence into a mel spectrum frame sequence, and perform speech waveform synthesis on the mel spectrum frame sequence to output synthesized speech.

[0007] Optionally, in a first implementation manner of the first aspect, the extracting the speaker embedding feature and the emotion embedding feature of the speech data comprises: performing noise reduction and dereverberation processing on the speech data to obtain preprocessed speech data; performing speech activity detection on the preprocessed speech data to obtain a speech start point and a speech end point; based on the speech start point and the speech end point, performing truncation on the preprocessed speech data to obtain valid speech data; extracting the mel spectrum of the valid speech data and inputting the valid speech data into a speaker recognition model for recognition, and taking the speaker embedding representation information output by the last hidden layer of the speaker recognition model as the speaker embedding feature; inputting the valid speech data into an emotion recognition model for recognition, and taking the emotion embedding representation information output by the last hidden layer of the emotion recognition model as the emotion embedding feature.

[0008] Optionally, in a second implementation manner of the first aspect, the training sample used by the speech semantic synthesis model comprises: a text word sequence sample, a speaker embedding feature sample, an emotion embedding feature sample and a speech semantic word sequence sample.

[0009] Optionally, in a third implementation manner of the first aspect, the generation method of the speech semantic word sequence sample comprises: obtaining an original speech sample and performing preprocessing; extracting the mel spectrum of the preprocessed speech sample; inputting the extracted mel spectrum into a speech semantic recognition model for recognition; taking the speech semantic representation information output by the last hidden layer of the speech semantic recognition model as the speech semantic word sequence sample.

[0010] Optionally, in a fourth implementation manner of the first aspect, the training method of the speech semantic synthesis model comprises: obtaining original training samples, the original training samples comprising text samples and speech samples; generating text word sequence samples based on the text samples; generate a speaker embedding feature sample, an emotion embedding feature sample and a speech semantic word segmentation sequence sample based on the speech sample; take the text word segmentation sequence sample, the speaker embedding feature sample and the emotion embedding feature sample as input of a pre-training model, take the speech semantic word segmentation sequence sample as output of the pre-training model, train the pre-training model to obtain a trained speech semantic synthesis model.

[0011] Optionally, in a fifth implementation manner of the first aspect, the speech synthesis method further includes: In the extraction of the emotion embedding features of the speech data, if there are multiple extracted emotion embedding features corresponding to the speech data, the emotion embedding features are clustered, and a preset number of emotion embedding features closest to the cluster center are selected; An average value of the selected emotion embedding features is calculated to obtain an average emotion embedding feature, and the average emotion embedding feature is taken as an emotion embedding feature input to the speech semantic synthesis model.

[0012] The second aspect of the present application provides a speech synthesis device, which includes: An acquisition module is configured to acquire text data and speech data to be processed; A preprocessing module is configured to convert the text data into a text word segmentation sequence, and extract speaker embedding features and emotion embedding features of the speech data; A speech semantic module is configured to input the text word segmentation sequence, the speaker embedding features and the emotion embedding features into a speech semantic synthesis model for processing, and output a speech semantic word segmentation sequence; A synthesis module is configured to convert the speech semantic word segmentation sequence into a mel-frequency spectrum frame sequence, and perform speech waveform synthesis on the mel-frequency spectrum frame sequence to output synthesized speech.

[0013] Optionally, in the first implementation manner of the second aspect, the preprocessing module is specifically configured to: perform noise reduction and dereverberation processing on the speech data to obtain preprocessed speech data; perform speech activity detection on the preprocessed speech data to obtain a speech start point and a speech end point; perform truncation on the preprocessed speech data based on the speech start point and the speech end point to obtain valid speech data; extract mel-frequency spectrum of the valid speech data and input speaker recognition model for recognition, and take speaker embedding representation information output by a last hidden layer of the speaker recognition model as the speaker embedding features; input the effective voice data into an emotion recognition model for recognition, and use emotion embedding representation information output by a last hidden layer of the emotion recognition model as the emotion embedding feature.

[0014] Optionally, in a second implementation manner of the second aspect, the training sample used by the voice semantic synthesis model comprises a text word segmentation sequence sample, a speaker embedding feature sample, an emotion embedding feature sample and a voice semantic word segmentation sequence sample.

[0015] Optionally, in a third implementation manner of the second aspect, the voice synthesis device further comprises: a sample generation module configured to acquire original voice samples and perform preprocessing, extract a mel spectrum of the preprocessed voice samples, input the extracted mel spectrum into a voice semantic recognition model for recognition, and use voice semantic representation information output by a last hidden layer of the voice semantic recognition model as a voice semantic word segmentation sequence sample.

[0016] Optionally, in a fourth implementation manner of the second aspect, the voice synthesis device further comprises: a training module configured to acquire original training samples, wherein the original training samples comprise text samples and voice samples, generate a text word segmentation sequence sample based on the text samples, generate a speaker embedding feature sample, an emotion embedding feature sample and a voice semantic word segmentation sequence sample based on the voice samples, use the text word segmentation sequence sample, the speaker embedding feature sample and the emotion embedding feature sample as input of a pre-training model, use the voice semantic word segmentation sequence sample as output of the pre-training model, train the pre-training model to obtain a trained voice semantic synthesis model.

[0017] Optionally, in a fifth implementation manner of the second aspect, the voice synthesis device further comprises: a clustering module configured to, when extracting emotion embedding features of the voice data, if there are multiple extracted emotion embedding features corresponding to the voice data, cluster the emotion embedding features, and select a preset number of emotion embedding features closest to a clustering center, calculate an average value of the selected emotion embedding features to obtain an average emotion embedding feature, and use the average emotion embedding feature as an emotion embedding feature input into the voice semantic synthesis model.

[0018] The third aspect of the present application provides a computer device, comprising a memory and at least one processor, the memory stores instructions; the at least one processor invokes the instructions in the memory, so that the computer device executes the voice synthesis method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described speech synthesis method.

[0020] The speech synthesis method provided by this invention extracts the emotion embedding features from speech data, transforming emotional information into a high-dimensional continuous feature space instead of relying on discrete labels. The large model can learn the essential acoustic features of emotion through these emotion embedding features. Even with scarce original data, it can still generate a continuous emotional intensity spectrum through feature vector interpolation, implicitly achieving data augmentation and solving the problem of imbalanced emotional data sample distribution in traditional speech synthesis methods that rely on internet emotional data. To achieve precise and controllable emotion, this invention also decouples the model input by separating the speaker embedding features and emotion embedding features from the speech data before inputting them into the model separately. This allows users to independently adjust the emotion type and intensity, avoiding control failure caused by timbre-emotion coupling; simultaneously, emotion expression can be directly manipulated based on feature vector space operations. Furthermore, to further enhance the emotional expressiveness of the synthesized speech, this invention converts text data into a text segmentation sequence and inputs it in conjunction with the emotion embedding features into the speech semantic synthesis model, generating a speech semantic segmentation sequence that integrates semantics and emotion. This invention first converts the synthesized speech semantic segmentation sequence into a Mel spectrum to enhance the acoustic details of emotion, and then outputs richly layered natural speech through a vocoder. This invention provides a highly controllable and expressive emotional speech synthesis solution for scenarios such as virtual digital humans and human-computer interaction. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of one embodiment of the speech synthesis method in this invention; Figure 2 This is a schematic diagram of one embodiment of the speech synthesis device in this invention; Figure 3 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0022] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the speech synthesis method in this invention includes: 101. Obtain the text and voice data to be processed; This embodiment specifically uses text data and speech data to complete speech synthesis. The text data is the text content that needs to be converted into speech, which can be sentences, paragraphs, or longer texts. Text data usually exists in the form of strings and may contain punctuation marks, numbers, special characters, etc.

[0024] Speech data consists of reference speech samples used to extract the vocal features of the target speaker. Speech data is typically stored in digital audio format. These reference speeches contain the vocal features of the target speaker and are used for subsequent speaker embedding feature extraction.

[0025] 102. Convert the text data into a text segmentation sequence, and extract the speaker embedding features and emotion embedding features from the speech data; In this embodiment, to address the issues of imbalanced emotional data, unstable emotional intensity, and insufficient emotional controllability, the text data is converted into a text segmentation sequence to achieve semantic and emotional fusion. Simultaneously, speaker embedding features and emotional embedding features are extracted from the speech data to achieve timbre and emotion separation in the speech data.

[0026] (1) Text segmentation: This involves dividing continuous text into meaningful linguistic units (such as words, subwords, or characters). For example: “Today the weather is very good” → [“Today”, “weather”, “very good”]. Specifically, rule-based, statistical, or deep learning-based segmentation techniques, such as Jieba (Chinese), WordPiece, and BERT segmenter, can be used for text segmentation to convert text data into a text segmentation sequence, that is, to convert text data into tokens (the basic units in text processing) to express features.

[0027] (2) Extracting speaker embedding features: Speaker embedding is a low-dimensional vector used to represent the speaker's voice features (such as timbre, tone quality, etc.). Pre-trained speaker recognition models, such as x-vector, d-vector, ECAPA-TDNN, cam++ speaker model, etc., can be used to extract speaker embedding features.

[0028] (3) Extracting emotion embedding features: Emotion embedding is a low-dimensional vector used to represent the emotional state in speech. Emotion recognition models (such as the wav2vec2 emotion vector model) can be used to extract emotion-related features in speech.

[0029] In one optional embodiment, before extracting speaker embedding features and emotion embedding features from the speech data, the speech data needs to be denoised and dedeveraged to obtain preprocessed speech data. Then, Voice Activity Detection (VAD) is performed on the preprocessed speech data to obtain the speech start point and speech end point. Finally, based on the speech start point and speech end point, the preprocessed speech data is truncated to obtain valid speech data, i.e., data containing speech content. This embodiment specifically uses the preprocessed valid speech data to extract speaker embedding features and emotion embedding features.

[0030] When extracting speaker embedding features, it is necessary to first extract the Mel spectrum of the valid speech data, and then input the extracted Mel spectrum into the speaker recognition model for recognition. The speaker embedding representation information output by the last hidden layer of the speaker recognition model is the speaker embedding feature to be extracted. This embodiment uses a speaker recognition model to extract sound features from speech data, thereby generating a fixed-dimensional vector representation. The extracted speaker embedding features can be used to enable synthesized speech to mimic the sound features of a specific speaker.

[0031] When extracting emotion embedding features, valid speech data is input into an emotion recognition model for identification. The emotion embedding representation information output by the final hidden layer of the emotion recognition model is the emotion embedding feature to be extracted. This embodiment extracts emotion-related features from speech data through an emotion recognition model and generates an emotion feature vector representation. Extracting emotion embedding features can be used to enable synthesized speech to express specific emotional nuances, enhancing expressiveness.

[0032] 103. Input the text segmentation sequence, the speaker embedding feature, and the emotion embedding feature into the speech semantic synthesis model for processing, and output the speech semantic segmentation sequence; In this embodiment, the speech semantic synthesis model preferably uses the Qwen2.5-0.5B structure as a pre-trained model, which is obtained through model training. The training samples used by the speech semantic synthesis model include: text segmentation sequence samples, speaker embedding feature samples, emotion embedding feature samples, and speech semantic segmentation sequence samples.

[0033] The inputs to the speech semantic synthesis model are: (1) a text segmentation sequence, which provides the text content to be synthesized; (2) a speaker embedding vector, which provides the voice features of the target speaker; and (3) an emotion embedding vector, which provides the target emotion features. The output of the speech semantic synthesis model is a speech semantic segmentation sequence, which is an intermediate data that integrates the semantic information of the text and the voice and emotion features of the speech.

[0034] In an optional embodiment, the following method is used to generate speech semantic segmentation sequence samples: Step 1: Obtain the original speech samples and perform preprocessing; In this optional embodiment, the original voice sample can be recorded voice data, or it can be a publicly available voice dataset such as LibriSpeech, Common Voice, TIMIT, etc., or it can be obtained by crawling voice content from podcasts, videos, and other media.

[0035] The original speech samples are resampled, specifically by unifying audio samples with different sampling rates to a target sampling rate (e.g., 16kHz). Then, denoising is performed, using traditional methods such as spectral subtraction and Wiener filtering, or deep learning-based denoising models (such as RNNoise and Demucs). Furthermore, speech activity detection (VAD) is performed on the denoised speech samples to detect and segment effective speech segments and remove silent parts.

[0036] Step 2: Extract the Mel spectrum of the preprocessed speech samples; Mel spectrum is a feature representation that maps the audio spectrum to a Mel scale, designed to simulate the human auditory system's perception of different sound frequencies. By converting a linear frequency axis to a non-linear Mel scale, equal intervals on the frequency axis correspond to the same distances perceived by the human ear, thus better reflecting the differences in how the human ear perceives sound frequencies.

[0037] First, the original audio signal needs to be sampled, framed, and windowed. Then, a Short-Time Fourier Transform (STFT) is performed on each frame of audio to convert the time-domain signal into a frequency-domain signal, obtaining the amplitude spectrum. This spectrum is then input into a Mel filter bank for processing. Applying the filter bank to the amplitude spectrum yields the energy of each filter. Finally, taking the logarithm of the energy of each filter gives the Mel spectrum.

[0038] Step 3: Input the extracted Mel spectrum into the speech semantic recognition model for recognition; Step 4: Use the speech semantic representation information output by the last hidden layer of the speech semantic recognition model as a speech semantic word segmentation sequence sample.

[0039] This optional embodiment preferably uses S3Tokenizer as the speech semantic recognition model. S3Tokenizer is a supervised semantic speech tokenizer based on a pre-trained SenseVoice-Large model. Its core function is to extract speech tokens with strong semantic relationships through deep learning, thereby enhancing the semantic representation capability of speech signals. The output of the last hidden layer of the speech semantic recognition model is speech semantic representation information, which is extracted and used as speech semantic word segmentation sequence samples.

[0040] In one optional embodiment, the training method for the speech semantic synthesis model includes: Step 1: Obtain the original training samples, which include text samples and speech samples; Step 2: Based on the text sample, generate a text segmentation sequence sample; Step 3: Based on the speech samples, generate speaker embedding feature samples, emotion embedding feature samples, and speech semantic segmentation sequence samples; Step 4: Using the text segmentation sequence samples, the speaker embedding feature samples, and the emotion embedding feature samples as inputs to the pre-trained model, and using the speech semantic segmentation sequence samples as the output of the pre-trained model, train the pre-trained model to obtain a trained speech semantic synthesis model.

[0041] In this optional embodiment, before training the model, it is necessary to convert the text samples in the original training samples into text segmentation sequence samples, and at the same time extract the speaker embedding features and emotion embedding features from the speech samples to obtain speaker embedding feature samples and emotion embedding feature samples, and generate speech semantic segmentation sequence samples based on the speech samples.

[0042] In this optional embodiment, the speech-semantic synthesis model preferably adopts the Qwen2.5-0.5B structure as the pre-training model. The pre-training model is trained by taking text segmentation sequence samples, speaker embedding feature samples and emotion embedding feature samples as inputs and speech-semantic segmentation sequence samples as outputs until the model parameters converge, thus obtaining the trained speech-semantic synthesis model.

[0043] 104. Convert the speech semantic segmentation sequence into a Mel spectrum frame sequence, and perform speech waveform synthesis on the Mel spectrum frame sequence to output synthesized speech.

[0044] In this embodiment, an acoustic model (such as a neural network) is preferably used to first map the speech semantic segmentation sequence into a Mel spectrum frame sequence. Each frame represents the sound spectrum features within a short time period (such as 10-25 milliseconds). Then, a vocoder is used to convert the frequency domain representation of the Mel spectrum frame sequence into a time domain audio waveform, which is a digital audio file (such as WAV format), and the synthesized speech can be played directly using a player.

[0045] The synthesized speech in this embodiment not only has text content that is identical to the input text, but also has the voice features of the target speaker (controlled by the speaker embedding vector) and the specified emotional state (controlled by the emotion embedding vector).

[0046] In an optional embodiment, in order to further improve the emotional stability of synthesized speech, when extracting emotional embedding features from speech data, if there are multiple emotional embedding features corresponding to the extracted speech data, then the emotional embedding features are clustered, and a preset number of emotional embedding features closest to the cluster center are selected; then the average value of the selected emotional embedding features is calculated to obtain the average emotional embedding feature, and the average emotional embedding feature is used as the emotional embedding feature of the input speech semantic synthesis model.

[0047] Using the average sentiment embedding feature as the sentiment embedding vector, along with the text segmentation sequence and speaker embedding vector, as input into the model can avoid the emotional fluctuations of a single sample (such as excessive or insufficient emotion in a certain speech segment). By integrating the common features of multiple speech data, it better reflects the typical expression of this sentiment category, and also improves the model's ability to adapt to unseen emotional expressions.

[0048] This embodiment extracts the emotion embedding features from speech data, transforming emotional information into a high-dimensional continuous feature space instead of relying on discrete labels. The large model can learn the essential acoustic features of emotion (such as fundamental frequency and energy distribution) through these emotion embedding features. Even with scarce original data, it can still generate a continuous spectrum of emotional intensity (such as "mild anger" to "intense anger") through feature interpolation, implicitly achieving data augmentation. This solves the problem of imbalanced emotional data sample distribution that traditional speech synthesis methods rely on internet emotional data.

[0049] To achieve precise and controllable emotion, this embodiment also decouples the model input by separating the speaker embedding features (timbre) and emotion embedding features (emotion) from the speech data and inputting them into the model. This allows users to independently adjust the emotion type and intensity, avoiding control failure caused by timbre-emotion coupling. Simultaneously, based on feature space operations, emotional expression can be directly manipulated through mathematical operations (such as feature addition, subtraction, and scaling). For example, by continuously adjusting the emotion intensity, complex emotional speech can be generated in real time.

[0050] To further enhance the emotional expressiveness of synthesized speech, this embodiment converts text data into a text segmentation sequence and inputs it into the speech semantic synthesis model in conjunction with emotion embedding features, generating a speech semantic segmentation sequence that integrates semantics and emotion. For example, the text "Great!" generates a high-pitched, fast-paced sequence under the emotion of "joy," and a low-pitched, prolonged sequence under the emotion of "sarcasm," ensuring a high degree of match between emotion and content. This embodiment first converts the synthesized speech semantic segmentation sequence into a Mel spectrum to enhance the acoustic details of emotion (such as energy concentration in anger and low-frequency enhancement in sadness), and then outputs richly layered natural speech through a vocoder. This invention provides a highly controllable and expressive emotional speech synthesis solution for scenarios such as virtual digital humans and human-computer interaction.

[0051] The speech synthesis method in the embodiments of the present invention has been described above. The speech synthesis apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the speech synthesis device in this invention includes: The acquisition module 201 is used to acquire the text data and voice data to be processed; The preprocessing module 202 is used to convert the text data into a text segmentation sequence and extract the speaker embedding features and emotion embedding features of the speech data; The speech semantic module 203 is used to input the text segmentation sequence, the speaker embedding feature and the emotion embedding feature into the speech semantic synthesis model for processing, and output the speech semantic segmentation sequence. The synthesis module 204 is used to convert the speech semantic segmentation sequence into a Mel spectrum frame sequence, and to perform speech waveform synthesis on the Mel spectrum frame sequence to output synthesized speech.

[0052] In an optional embodiment, the preprocessing module 202 is further configured to: The speech data is subjected to noise reduction and dereverberation processing to obtain preprocessed speech data; The preprocessed speech data is subjected to speech activity detection to obtain the speech start point and speech end point; Based on the start point and end point of the speech, the preprocessed speech data is truncated to obtain valid speech data; The Mel spectrum of the effective speech data is extracted and input into the speaker recognition model for recognition. The speaker embedding representation information output by the last hidden layer of the speaker recognition model is used as the speaker embedding feature. The effective speech data is input into the emotion recognition model for recognition, and the emotion embedding representation information output by the last hidden layer of the emotion recognition model is used as the emotion embedding feature.

[0053] In an optional embodiment, the training samples used by the speech semantic synthesis model include: text segmentation sequence samples, speaker embedding feature samples, emotion embedding feature samples, and speech semantic segmentation sequence samples.

[0054] In an optional embodiment, the speech synthesis device further includes: The sample generation module is used to acquire the original speech samples and preprocess them; extract the Mel spectrum of the preprocessed speech samples; input the extracted Mel spectrum into the speech semantic recognition model for recognition; and use the speech semantic representation information output by the last hidden layer of the speech semantic recognition model as the speech semantic word segmentation sequence sample.

[0055] In an optional embodiment, the speech synthesis device further includes: The training module is used to acquire original training samples, which include text samples and speech samples; based on the text samples, it generates text segmentation sequence samples; based on the speech samples, it generates speaker embedding feature samples, emotion embedding feature samples, and speech semantic segmentation sequence samples; using the text segmentation sequence samples, speaker embedding feature samples, and emotion embedding feature samples as inputs to the pre-trained model, and using the speech semantic segmentation sequence samples as outputs to train the pre-trained model, thereby obtaining a trained speech semantic synthesis model.

[0056] In an optional embodiment, the speech synthesis device further includes: The clustering module is used to, when extracting the emotion embedding features of the speech data, if there are multiple extracted speech data corresponding to emotion embedding features, then cluster each emotion embedding feature and select a preset number of emotion embedding features closest to the cluster center; calculate the average value of each selected emotion embedding feature to obtain the average emotion embedding feature, and use the average emotion embedding feature as the emotion embedding feature input to the speech semantic synthesis model.

[0057] Since the embodiments of the device part correspond to the embodiments of the above method, the description of the speech synthesis device provided by the present invention should refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above speech synthesis method.

[0058] above Figure 2 The speech synthesis device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The computer device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0059] Figure 3This is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computer device 500.

[0060] Computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0061] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the speech synthesis method in the above embodiments. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, the computer-readable storage medium storing instructions, which, when executed on a computer, cause the computer to perform the steps of the speech synthesis method.

[0062] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0063] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, The speech synthesis method includes: Acquire the text and voice data to be processed; The text data is converted into a text segmentation sequence, and the speaker embedding features and emotion embedding features of the speech data are extracted. The text segmentation sequence, the speaker embedding feature, and the emotion embedding feature are input into the speech semantic synthesis model for processing, and the speech semantic segmentation sequence is output. The speech semantic segmentation sequence is converted into a Mel spectrum frame sequence, and the Mel spectrum frame sequence is used to synthesize speech waveforms to output synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that, The extraction of speaker embedding features and emotion embedding features from the speech data includes: The speech data is subjected to noise reduction and dereverberation processing to obtain preprocessed speech data; The preprocessed speech data is subjected to speech activity detection to obtain the speech start point and speech end point; Based on the start point and end point of the speech, the preprocessed speech data is truncated to obtain valid speech data; The Mel spectrum of the effective speech data is extracted and input into the speaker recognition model for recognition. The speaker embedding representation information output by the last hidden layer of the speaker recognition model is used as the speaker embedding feature. The effective speech data is input into the emotion recognition model for recognition, and the emotion embedding representation information output by the last hidden layer of the emotion recognition model is used as the emotion embedding feature.

3. The speech synthesis method according to claim 1, characterized in that, The training samples used in the speech semantic synthesis model include: text segmentation sequence samples, speaker embedding feature samples, emotion embedding feature samples, and speech semantic segmentation sequence samples.

4. The speech synthesis method according to claim 3, characterized in that, The method for generating the speech semantic word segmentation sequence samples includes: Obtain the original speech samples and perform preprocessing; Extract the Mel spectrum of the preprocessed speech samples; The extracted Mel spectrum is input into the speech semantic recognition model for recognition; The speech semantic representation information output by the last hidden layer of the speech semantic recognition model is used as a speech semantic word segmentation sequence sample.

5. The speech synthesis method according to any one of claims 1-4, characterized in that, The training method for the speech semantic synthesis model includes: Obtain the original training samples, which include text samples and speech samples; Based on the text sample, generate a text segmentation sequence sample; Based on the speech samples, speaker embedding feature samples, emotion embedding feature samples, and speech semantic word segmentation sequence samples are generated. The pre-trained model is trained by using the text segmentation sequence samples, the speaker embedding feature samples, and the emotion embedding feature samples as inputs and the speech semantic segmentation sequence samples as outputs, thereby obtaining a trained speech semantic synthesis model.

6. The speech synthesis method according to claim 1, characterized in that, The speech synthesis method further includes: When extracting the emotional embedding features of the speech data, if there are multiple emotional embedding features corresponding to the extracted speech data, then the emotional embedding features are clustered and a preset number of emotional embedding features closest to the cluster center are selected. The average value of each selected emotion embedding feature is calculated to obtain the average emotion embedding feature, and the average emotion embedding feature is used as the emotion embedding feature input to the speech semantic synthesis model.

7. A speech synthesis device, characterized in that, The speech synthesis device includes: The acquisition module is used to acquire the text and voice data to be processed. The preprocessing module is used to convert the text data into a text segmentation sequence and extract the speaker embedding features and emotion embedding features of the speech data; The speech semantic module is used to input the text segmentation sequence, the speaker embedding feature and the emotion embedding feature into the speech semantic synthesis model for processing, and output the speech semantic segmentation sequence. The synthesis module is used to convert the speech semantic segmentation sequence into a Mel spectrum frame sequence, and to synthesize the speech waveform from the Mel spectrum frame sequence to output synthesized speech.

8. The speech synthesis device according to claim 7, characterized in that, The preprocessing module is also used for: The speech data is subjected to noise reduction and dereverberation processing to obtain preprocessed speech data; The preprocessed speech data is subjected to speech activity detection to obtain the speech start point and speech end point; Based on the start point and end point of the speech, the preprocessed speech data is truncated to obtain valid speech data; The Mel spectrum of the effective speech data is extracted and input into the speaker recognition model for recognition. The speaker embedding representation information output by the last hidden layer of the speaker recognition model is used as the speaker embedding feature. The effective speech data is input into the emotion recognition model for recognition, and the emotion embedding representation information output by the last hidden layer of the emotion recognition model is used as the emotion embedding feature.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the computer device to perform the speech synthesis method as described in any one of claims 1-6.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the speech synthesis method as described in any one of claims 1-6.

Images