A speech synthesis method, device, computer equipment and storage medium

Abstract

The application belongs to the field of artificial intelligence, medical treatment and finance, and relates to a speech synthesis method, which comprises the following steps: phoneme conversion and context embedding are performed on a text data set to obtain a phoneme sequence and a text context vector; reference speech features of a reference audio sample are extracted; the phoneme-level prior latent features are generated by inputting the phoneme-level prior latent features into a phoneme-level variational auto-encoding model; the reconstructed speech features are obtained; target speech features of a target audio sample are extracted; the target speech features are aligned with the phoneme sequence and are up-sampled to obtain target audio frame-level features; and a conditional control speech synthesis model is trained based on the reconstructed speech features and the target audio frame-level features to synthesize speech for a text to be converted. The application also provides a speech synthesis device, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the text to be converted can be stored in the blockchain. The application improves speaker fidelity and text intelligibility.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology, and digital medical technology, and in particular to a speech synthesis method, apparatus, computer equipment, and storage medium. Background Art

[0002] In the fields of medicine and finance, accurate information transmission is crucial. Text-to-speech (TTS) technology, as an effective means of information transmission, has made significant technological progress in recent years. TTS models are designed to accurately replicate the diversity of human speech, including the unique speaker identity and nuances of language. This is crucial for enhancing the authenticity and relatability of information transmission. In particular, in scenarios such as medical diagnostic reports and financial data interpretation, converting text into natural, fluent speech through TTS technology can significantly improve information accessibility and comprehension.

[0003] However, despite significant progress in mimicking the richness of human speech, achieving an optimal balance between speaker fidelity and text comprehensibility in the face of diverse control requirements remains a major challenge. Human speech is characterized by its complexity and variability, including unique speaker identities, varying speech rhythms, intonations, and language styles. These characteristics place extremely high demands on the synthesis capabilities of TTS models.

[0004] An ideal TTS model should capture essential speaker characteristics, such as timbre, speaking style, accent, and emotion, to achieve high-fidelity speech synthesis. At the same time, the model must ensure that the synthesized speech is clear and understandable, maintaining strong text comprehensibility to meet the basic requirements of information transmission. However, in practical applications, there is often a certain conflict between speaker fidelity and text comprehensibility.

[0005] To date, most TTS research has relied on metrics such as speaker similarity, naturalness (MOS), and word error rate (WER) to assess speaker fidelity and text comprehensibility. However, relatively little research has explored how to independently control each element when these two factors conflict. Therefore, maintaining high speaker fidelity while ensuring speech clarity and text comprehensibility has become a key challenge in the current field of TTS technology. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, computer device and storage medium to solve the technical problem in the prior art that it is difficult to maintain a perfect balance between speaker fidelity and text comprehensibility.

[0007] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solutions:

[0008] Acquire a text dataset and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0009] Inputting the text dataset and the audio sample set into a pre-trained speech synthesis model for training, the speech synthesis model comprising a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0010] Inputting the text data set into the phoneme conversion layer and the context embedding layer for processing respectively, and obtaining a phoneme sequence and a text context vector respectively;

[0011] Extracting NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0012] Inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature;

[0013] Inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features;

[0014] Extracting NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and aligning the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features;

[0015] Upsampling the phoneme alignment features to the frame level to obtain target audio frame-level features;

[0016] Iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model;

[0017] The text to be converted is obtained, and the text to be converted is input into the conditionally controlled speech synthesis model to obtain the target synthesized speech.

[0018] In order to solve the above technical problems, the present application also provides a speech synthesis device, which adopts the following technical solution:

[0019] An acquisition module, configured to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0020] An input module, configured to input the text dataset and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0021] A conversion and embedding module, configured to input the text dataset into the phoneme conversion layer and the context embedding layer for processing, thereby obtaining a phoneme sequence and a text context vector respectively;

[0022] A feature extraction module, configured to extract NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0023] an encoding module, configured to input the phoneme sequence, the text context vector, and the reference speech feature into the phoneme-level latent diffusion model, and fuse the reference condition and the text condition to generate a phoneme-level priori latent feature;

[0024] A decoding module, configured to input the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features;

[0025] An alignment module is configured to extract NANSY features of the target audio sample using the NANSY synthesizer to obtain target speech features, and align the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features;

[0026] An upsampling module, configured to upsample the phoneme alignment features to a frame level to obtain target audio frame-level features;

[0027] An iteration module, configured to iteratively train the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model;

[0028] The speech synthesis module is used to obtain the text to be converted, input the text to be converted into the conditionally controlled speech synthesis model, and obtain the target synthesized speech.

[0029] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0030] The computer device includes a memory and a processor. The memory stores computer-readable instructions. When the processor executes the computer-readable instructions, the steps of the speech synthesis method described above are implemented.

[0031] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0032] The computer readable storage medium stores computer readable instructions, which are executed by the processor to implement the steps of the speech synthesis method.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] The present application provides a speech synthesis method, which converts phonemes and context embedding of text data set to obtain corresponding phoneme sequence and text context vector, extracts NANSY features of reference audio sample, combines phoneme sequence and text context vector of input text, generates target speech by using phoneme-level latent diffusion model and phoneme-level variational auto-encoding model, skillfully fuses reference condition and text condition, accurately expresses text content while retaining speaker style, and can generate natural and fluent speech with rich expression, while maintaining good timbre similarity and content accuracy, improving speech synthesis quality, and significantly improving the balance between speaker fidelity and text intelligibility. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the scheme in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0036] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0037] Figure 2 is a flowchart of one embodiment of the speech synthesis method according to the present application;

[0038] Figure 3 is a structural schematic diagram of one embodiment of the speech synthesis device according to the present application;

[0039] Figure 4 is a structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0041] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0044] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0046] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0047] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the speech synthesis device is generally set in the server / terminal device.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0049] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application, comprising the following steps:

[0050] Step S201: Acquire a text dataset and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample.

[0051] Collect audio sample sets. Audio sample sets can be collected using professional recording studio equipment or through public data platforms. Preprocess the collected raw audio sample data, including removing noise and adjusting audio quality, to obtain a preprocessed audio sample set. Obtain the text dataset corresponding to the audio sample set and annotate the audio sample set with text. Match the corresponding text with the corresponding audio clips to establish a mapping relationship between text and audio, providing supervised learning data for subsequent model training.

[0052] In this embodiment, the electronic device on which the speech synthesis method is running (eg Figure 1The server / terminal device shown in the figure can receive the acquired text dataset and audio sample set via a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0053] The audio sample set includes a large number of reference audio samples and target audio samples, that is, a reference audio sample subset and a target audio sample subset. The reference audio sample subset is a reference speech sampled from the target audio sample subset. The audio sample set can include speech samples with different accents and dialects.

[0054] For example, in the financial field, the audio sample set can be audio data containing financial professional terms, financial market data, stock trading information, financial reports, etc., or it can be recordings of conversations between financial customer service and customers; in the medical field, the audio sample set can be audio data containing medical diagnoses, treatment plans, drug names, disease names, etc., or it can be audio data containing medical guidance information such as drug instructions, health check instructions, disease prevention suggestions, etc., or it can be recordings of conversations between doctors and patients of different age groups and genders during the diagnosis and treatment process.

[0055] Step S202: Input the text dataset and the audio sample set into a pre-trained speech synthesis model for training. The speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model.

[0056] In this embodiment, the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model and a phoneme-level variational autoencoder model. The speech synthesis model is trained through two stages, namely, VAE (variational autoencoder model) training and LDM (latent diffusion model) training. The input text is converted into a phoneme sequence at the phoneme level through the phoneme conversion layer, and the input text is converted into a text context vector through the context embedding layer. VAE training reconstructs the NANSY features of the reference audio sample based on the given target audio sample and phoneme sequence, and LDM training generates VAE latent features based on the given input text and reference audio sample.

[0057] In this embodiment, before the audio sample set is input into the speech synthesis model, it is first preprocessed, including denoising, normalization and other operations. Specifically, a wavelet denoising algorithm is used to reduce the noise of the speech signals in the audio sample set to remove background noise interference; then, the speech signals in the denoised audio sample set are normalized to map the amplitude of the speech signals to the interval [-1, 1] to obtain a standardized audio sample set.

[0058] In step S203, the text data set is input into the phoneme conversion layer and the context embedding layer for processing, thereby obtaining a phoneme sequence and a text context vector.

[0059] Specifically, the text dataset is input into the phoneme conversion layer, and the text is mapped into phonemes by adopting the grapheme-to-phoneme conversion model to obtain a phoneme sequence; the text dataset is input into the context embedding layer, and the pre-trained XLM-RoBERTa model is used to perform context embedding vector representation on the text dataset to obtain a text context vector.

[0060] The phoneme conversion layer converts text into its corresponding pronunciation. Phonemes are symbolic representations of speech sounds. Converting text to phonemes helps the model accurately learn the phoneme mapping of text. Specifically, the phoneme conversion layer predicts the corresponding phoneme sequence by capturing local and global patterns in the input sequence. The contextual embedding layer uses the XLM-RoBERTa model to convert the input text dataset into a vector representation containing contextual information for each word or subword in the text.

[0061] By converting text into phonemes and embedding its context, it helps to capture the phonetic features in the text and enhance the expressiveness of speech synthesis. It also helps to more accurately understand the meaning and contextual relationships of the text, improve the semantic accuracy of speech synthesis, and thus enhance the naturalness and fluency of speech synthesis.

[0062] In some optional implementations of this embodiment, the glyph-to-phoneme conversion model is a G2P model. Specifically, the input text data set is preprocessed, including operations such as word segmentation and part-of-speech tagging, to extract text features such as keywords and phrases; according to preset rules and templates, the extracted text features are converted into parameterized representations to obtain text feature parameters; the text feature parameters are input into the pre-trained G2P model, and the phoneme representation vector corresponding to each text feature parameter is obtained through forward propagation calculation; a beam search-based decoding algorithm is used to decode the phoneme representation vector output by the G2P model, converting it into a phoneme symbol sequence to obtain a complete phoneme sequence representation; the generated phoneme sequence is post-processed, such as merging and deleting phonemes, to optimize the fluency and naturalness of the phoneme sequence; and the optimized phoneme sequence is used for subsequent operations.

[0063] In some optional implementations of this embodiment, the XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer; the step of using the pre-trained XLM-RoBERTa model to represent the context embedding vector of the text dataset to obtain the text context vector includes:

[0064] The text samples in the text dataset are converted into vectors through the embedding layer to obtain text embedding vectors;

[0065] The text embedding vector is input into the Transformer encoder layer for encoding and transformation, and the text context vector is output.

[0066] In this embodiment, the text samples in the text dataset are respectively word-embedded (WordEmbedding), position-embedded (Position Embedding) and segment-embedded (Segment Embedding) through the embedding layer to obtain the corresponding word embedding vector, position embedding vector and segment embedding vector, and the word embedding vector, position embedding vector and segment embedding vector are concatenated to obtain the text embedding vector. The Transformer encoder layer is composed of multiple encoders stacked together, each encoder including multiple self-attention mechanisms and feedforward neural networks for encoding and transforming the input text embedding vector, wherein each layer of encoder receives the output of the previous layer of encoder as input and outputs a series of new vector representations, namely the text context vector.

[0067] In this embodiment, the XLM-RoBERTa model can also learn to switch between different languages.

[0068] Contextual embedding through the XLM-RoBERTa model can convert the input text into a high-dimensional vector representation, thereby more accurately capturing the semantic features of the text and helping to enhance the comprehensibility of the text.

[0069] Step S204: extract NANSY features of the reference audio sample using a NANSY synthesizer to obtain reference speech features.

[0070] In this embodiment, a pre-trained NANSY synthesizer is used to extract NANSY features of the reference audio sample. The NANSY features include language features, fundamental frequency (f0) features, periodic amplitude features, and non-periodic amplitude features.

[0071] Among them, the NANSY synthesizer mainly consists of two parts: the analysis module and the synthesis module. The analysis module is used for speech feature extraction and can extract key feature information from the input speech signal reference audio sample; the synthesis module is used for speech synthesis. Based on the features extracted by the analysis module, the synthesis module can generate high-quality speech output.

[0072] Step S205 : Input the phoneme sequence, text context vector, and reference speech features into a phoneme-level latent diffusion model, and fuse the reference conditions and text conditions to generate phoneme-level priori latent features.

[0073] In this embodiment, the phoneme-level latent diffusion model is a pre-trained model that includes a phoneme encoder, a context encoder, a retrieval encoder, a conditioner, and a latent diffusion encoder with dual classifier-free guidance (CFG). The phoneme-level latent diffusion model uses dual classifier-free guidance to enhance speaker fidelity and text comprehensibility.

[0074] Specifically, the phoneme sequence is encoded through a phoneme encoder to obtain a phoneme feature vector; the phoneme feature vector and the text context vector are input into the context encoder for encoding to obtain a text phoneme joint feature vector; the reference speech feature is encoded through a retrieval encoder to obtain a speech style feature vector; the text phoneme joint feature vector and the speech style feature vector are input into the conditioner, the reference condition and the text condition are fused for encoding to obtain a conditional feature vector; the conditional feature vector is weighted by a latent diffusion encoder to generate a phoneme-level prior latent feature.

[0075] In this embodiment, the conditioner and the dual classifier-free latent diffusion encoder are the two main components. The conditioner generates phoneme-level conditional information based on the reference speaker and input text. The conditioner shares the input from the context encoder. The context encoder is a Transformer encoder structure using a cross-attention mechanism to simulate context-aware features obtained from the output of the phoneme encoder and the text context vector.

[0076] In some optional implementations, to reduce the mismatch between training and inference, noise is added to the reference audio sample before NANSY feature extraction, and the sample is cut into random lengths to obtain a noisy reference audio sample, and NANSY features are extracted from the noisy reference audio sample.

[0077] In this embodiment, the reference speech features are input into the retrieval encoder, and weight calculation is performed through the cross-attention mechanism of the retrieval encoder. The query vector of the cross-attention mechanism of the retrieval encoder is a fixed-length token, which can be called a prototype. Therefore, the output of the retrieval encoder is also a fixed-length token, which encapsulates the speaker style of the reference audio sample.

[0078] The conditioner uses the speaker style and input text to encode speaker-related conditions and text content-related conditions, and obtains a conditional feature vector of speaker-aware phoneme-level conditions. Based on the obtained conditional feature vector, the latent features are obtained by adjusting the highly controllable CFG weights through the latent diffusion encoder.

[0079] The phoneme-level latent diffusion model is used to fuse reference conditions and text conditions to generate phoneme-level prior latent features, thereby enhancing speaker fidelity and text comprehensibility. The retrieval encoder is used to capture the speaker's style in the reference speech features, achieving zero-sample speech synthesis capability, greatly reducing computing requirements, and improving model training efficiency. The CFG weights are adjusted through the latent diffusion encoder to achieve high controllability of the dual CFG, thereby improving the balance between speaker fidelity and text comprehensibility.

[0080] In some optional implementations of this embodiment, the step of inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining the conditional feature vector includes:

[0081] Inputting the text phoneme joint feature vector and the speech style feature vector into a conditioner, wherein the conditioner includes a reference conditioner and a text conditioner;

[0082] The style condition feature vector is obtained by fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner to encode the style feature.

[0083] Perform text feature encoding on the text-phoneme joint feature vector through a text conditioner to obtain a text-condition feature vector;

[0084] The style conditional feature vector and the text conditional feature vector are output as conditional feature vectors.

[0085] The goal of the reference conditioner is to generate speaker-aware phoneme-level conditions by utilizing the speaker tag, i.e., the speech style feature vector output by the retrieval encoder, as the value of the cross-attention and using prototypes of the same length as the retriever encoder to encode speaker-related conditions, i.e., obtain the style condition feature vector. The text conditioner relies only on text input, i.e., the text-phoneme joint feature vector, and obtains the text condition feature vector by encoding conditions related to the text content.

[0086] By performing conditional encoding through the reference conditioner and the text conditioner respectively, the conditions of different aspects of speech synthesis are decoupled and separated, the conditions related to the reference speech and the input text can be independently controlled, the speaker fidelity and the text intelligibility have extraordinary control ability, and the flexibility of adjusting the speaker fidelity or the text intelligibility is improved.

[0087] In the embodiment, the architecture of the latent diffusion encoder is based on the Transformer encoder structure, specifically, the latent diffusion encoder includes an embedding layer, a plurality of stacked encoders and two MLP layers, and a conditional mechanism is simply added after the two MLP layers.

[0088] The style condition feature vector and the text condition feature vector are input into the latent diffusion encoder, first, the CFG (control free generation) weight of the style condition feature vector is adjusted to adjust the similarity to the speaker, and the CFG weight of the text condition feature vector is adjusted to control the fine intelligibility of the text.

[0089] In some optional implementation manners, the phoneme encoder, the context encoder, the retrieval encoder, the reference conditioner, the text conditioner and the latent diffusion encoder all adopt the Transformer encoder architecture, the number of encoders in the Transformer encoder architecture is set according to actual needs, and is not limited herein.

[0090] In the embodiment, the phoneme-level latent diffusion model is trained using the L1 loss, and the formula is as follows:

[0091]

[0092] In the formula, ∈ represents added noise, ∈ θ represents a diffusion model with parameters, μ is the mean value estimated by the phoneme-level variational auto-encoding model, t represents a time step, and represents the noise coefficient corresponding to the time t. In the application, random discarding of the output c text of the text conditioner and the output c spk of the reference conditioner is implemented to use the CFG weight during inference.

[0093] Specifically, it is assumed that c text is discarded by a probability of 5%, c spk is discarded by a probability of 10%, and both are additionally applied with a 10% dropout to promote the frequency of the empty condition scenario. The training uses discrete integer diffusion time steps and noise schedules. t is uniformly sampled from [1, T], where T = 200. Following the method of the previous diffusion model, a linear variance schedule is adopted, defined as β i = β1+(β T- b1)(i - 1) / (T - 1), with b1= 0.0001 and b2= 0.03. The noise coefficient T The formula for calculating the noise coefficient The formula for calculating the noise coefficient

[0094] In this embodiment, the latent diffusion encoder adopts double CFG, which can be expressed as follows:

[0095]

[0096] wherein, denotes the noise without classifier guidance, z t is the latent feature at time step t, defined as and Ф denotes a zero tensor corresponding to the empty conditional state, ω spk denotes the CFG weight of c spk , and ω text denotes the CFG weight of c text .

[0097] By using the phoneme-level latent diffusion model to perform more detailed operations on speech synthesis, the problem of balancing text and speaker similarity is directly solved, and the intelligibility and controllability of the model are enhanced.

[0098] In step S206, the phoneme-level prior latent feature is input into the phoneme-level variational auto-encoding model for decoding to obtain reconstructed speech features.

[0099] The phoneme-level variational auto-encoding model (VAE) processes the phoneme-level prior latent feature output by the phoneme-level latent diffusion model to reconstruct the NANSY feature of the reference audio sample.

[0100] In this embodiment, the phoneme-level variational auto-encoding model includes a latent feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder.

[0101] Specifically, the phoneme-level prior latent feature is input into the latent feature decoder, the duration predictor, the phoneme prosody decoder, and the upsampler, respectively; the phoneme-level prior latent feature is subjected to phoneme-level duration prediction by the duration predictor to obtain an audio predicted duration, and the audio predicted duration is input into the latent feature decoder; the phoneme-level prior latent feature and the audio predicted duration are decoded by the latent feature decoder to obtain predicted phoneme time sequence decoding features; the predicted phoneme time sequence decoding features and the phoneme-level prior latent feature are fused and decoded by the phoneme prosody decoder to obtain predicted phoneme prosody features; the predicted phoneme prosody features and the phoneme-level prior latent feature are fused and upsampled by the upsampler to obtain frame-level features; and the frame-level features are input into the frame-level decoder for decoding to obtain reconstructed speech features.

[0102] In this embodiment, the decoding process starts with decoding the phoneme-level prior latent features through a latent feature decoder. The phoneme-level prior latent features are input in parallel to the duration predictor, the phoneme prosody decoder and the upsampler. The duration predictor is responsible for estimating the duration of the phoneme level, and the phoneme prosody decoder is responsible for estimating the prosodic features of the phoneme level. The phoneme fundamental frequency (f0) can be obtained based on the phoneme prosodic features.

[0103] The architecture of the upsampler is almost the same as that in Parallel Tacotron 2, but does not include the channel dimension. The frame-level features upsampled by the upsampler are input to the frame-level decoder for decoding to reconstruct the NANSY features of the reference audio sample and obtain the reconstructed speech features.

[0104] The phoneme-level prior latent features are decoded by the phoneme-level variational autoencoder model to obtain reconstructed speech features. The information in the latent features can be used to accurately reconstruct the speech features. The reconstruction process can retain the important features of the reference audio signal, further improving the quality of speech synthesis and enhancing the accuracy of speech recognition.

[0105] Step S207: extract NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and use a pre-trained aligner to align the target speech features with the phoneme sequence to obtain phoneme alignment features.

[0106] During speech synthesis model training, target audio samples are introduced to guide NANSY feature reconstruction of reference audio samples. A pre-trained NANSY synthesizer is used to extract NANSY features from the target audio samples corresponding to the reference audio samples to obtain target speech features. The phoneme sequence and target speech features are then fed into a posterior encoder introduced for model training to encode them, obtaining phoneme-level posterior latent features. These phoneme-level posterior latent features are then fed into a latent feature decoder. A pre-trained aligner is used to align the target speech features with the phoneme sequence to obtain phoneme alignment features.

[0107] In this embodiment, the step of using a pre-trained aligner to align the target speech feature with the phoneme sequence to obtain the phoneme alignment feature includes:

[0108] Obtain a phoneme feature vector obtained by encoding the phoneme sequence by a phoneme encoder;

[0109] The target speech features are aligned with the phoneme feature vectors using a monotonic alignment search method through the aligner to obtain the phoneme alignment features.

[0110] Specifically, an alignment matrix of size M*N is created by the aligner, where M represents the length of the target speech feature vector and N represents the length of the phoneme sequence vector. The alignment matrix is ​​then initialized. A dynamic programming algorithm is used to traverse the alignment matrix and calculate the value of each element based on a monotone alignment search method. If the current target speech feature vector element matches the current phoneme sequence vector element, the value of the current matrix element is the value of the upper-left element, and 1 is added; otherwise, the value of the current matrix element is the maximum of the left and upper element values. Backtracking from the lower-right element of the alignment matrix, the optimal alignment path is obtained. If the value of the current matrix element is equal to the upper-left element value, 1 is added, and the current target speech feature vector element is aligned with the current phoneme sequence vector element and moved to the upper left corner. Otherwise, if the value of the current matrix element is equal to the upper element value, the element is moved upwards; if the value of the current matrix element is equal to the left element value, the element is moved leftwards. Based on the alignment path obtained by backtracking, the alignment relationship between the target speech feature and the phoneme sequence is determined, and phoneme-level alignment information, i.e., the phoneme alignment feature, is obtained.

[0111] By aligning the target speech features with the phoneme sequence, the accuracy of feature matching can be improved, the generalization ability of the model can be enhanced, and the interpretability and robustness of the model can be improved.

[0112] Step S208 : upsample the phoneme alignment features to the frame level to obtain target audio frame-level features.

[0113] Specifically, the phoneme alignment feature is input into the duration predictor to obtain the true duration of the audio; the true duration of the audio is input into the latent feature decoder, and the phoneme-level posterior latent feature and the true duration of the audio are decoded by the latent feature decoder to obtain the target phoneme timing feature; the target phoneme timing feature and the phoneme-level posterior latent feature are input into the phoneme prosody decoder for decoding to obtain the target phoneme prosody feature; the target phoneme prosody feature and the phoneme-level posterior latent feature are fused by the upsampler for upsampling to obtain the target audio frame-level feature.

[0114] Step S209: iteratively train the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until the iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model.

[0115] In this embodiment, according to the preset loss function, the loss value is calculated based on the reconstructed speech features and the target audio frame-level features, and the Adam optimizer is used to adjust the model parameters of the speech synthesis model according to the loss value. The adjusted model is iteratively trained until the iteration stop condition is met, that is, the number of iterations reaches the preset number or the first loss value does not change significantly, and a conditionally controlled speech synthesis model that has completed training is obtained.

[0116] In some optional implementations, the preset loss function includes one or more combinations of NANSY feature reconstruction loss, phoneme-level fundamental frequency reconstruction loss, duration loss, KL divergence of latent variables, and adversarial loss. Among them, NANSY feature reconstruction loss represents the loss between the reconstructed speech features and the target speech features; phoneme-level fundamental frequency reconstruction loss represents the loss between the predicted phoneme prosodic features and the target phoneme prosodic features; duration loss represents the loss between the predicted audio duration and the actual audio duration. Adversarial loss is trained by adding a discriminator. Specifically, the discriminator consists of a simple convolutional network and is trained using least squares loss and feature matching loss.

[0117] When the preset loss function is a combination of multiple combinations, the comprehensive loss value is obtained by weighted summing up each loss as the training loss.

[0118] Step S210: obtaining the text to be converted, inputting the text to be converted into a conditional control speech synthesis model to obtain the target synthesized speech.

[0119] In this embodiment, the acquired text input condition to be converted is used to control the speech synthesis model for inference to generate the target synthesized speech. During the inference process, a phoneme-level latent prior variable is generated by iterative denoising using a phoneme-level latent diffusion model. Specifically, a fast sampling method is adopted, using a variance noise plan such as [1e-4, 5e-4, 1e-3, 5e-3, 0.01, 0.02, 0.05, 0.2, 0.3, 0.5, 0.4, 0.3, 0.3, 0.2, 0.1, 0.1]. The generated phoneme-level prior latent variable is then processed by a phoneme-level variational autoencoder model, including upsampling to the NANSY frame level and estimating NANSY features. Finally, a speech waveform is synthesized using the estimated NANSY features by a NANSY synthesizer to obtain the target synthesized speech.

[0120] It should be emphasized that in order to further ensure the privacy and security of the text to be converted, the above text to be converted can also be stored in a node of a blockchain.

[0121] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0122] The application obtains corresponding phoneme sequences and text context vectors by phoneme conversion and context embedding on a text data set, extracts NANSY features of a reference audio sample, combines the phoneme sequences and text context vectors of the input text, and generates target speech by using a phoneme-level latent diffusion model and a phoneme-level variational auto-encoding model, thereby ingeniously fusing reference conditions and text conditions, accurately expressing text content while retaining speaker style, generating natural and fluent speech with rich expression, maintaining good timbre similarity and content accuracy, improving speech synthesis quality, and significantly improving the balance between speaker fidelity and text intelligibility.

[0123] Embodiments of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain optimal results.

[0124] Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0125] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0126] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0127] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 3 As shown, the speech synthesis device 300 of this embodiment includes: an acquisition module 301, an input module 302, a conversion and embedding module 303, a feature extraction module 304, an encoding module 305, a decoding module 306, an alignment module 307, an upsampling module 308, an iteration module 309, and a speech synthesis module 310. Among them:

[0129] The acquisition module 301 is used to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0130] The input module 302 is used to input the text dataset and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0131] The conversion embedding module 303 is used to input the text data set into the phoneme conversion layer and the context embedding layer for processing, thereby obtaining a phoneme sequence and a text context vector respectively;

[0132] The feature extraction module 304 is used to extract NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0133] The encoding module 305 is used to input the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, and fuse the reference condition and the text condition to generate a phoneme-level priori latent feature;

[0134] The decoding module 306 is configured to input the phoneme-level prior latent feature into the phoneme-level variational auto-encoding model for decoding to obtain reconstructed speech features;

[0135] The alignment module 307 is configured to extract NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and align the target speech features with the phoneme sequence by using a pre-trained aligner to obtain phoneme alignment features.

[0136] The up-sampling module 308 is configured to up-sample the phoneme alignment features to a frame level to obtain target audio frame-level features.

[0137] The iteration module 309 is configured to iteratively train the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stopping condition is met to obtain a final conditional control speech synthesis model.

[0138] The speech synthesis module 310 is configured to obtain a text to be converted, input the text to be converted into the conditional control speech synthesis model, and obtain target synthesized speech.

[0139] It should be emphasized that, in order to further ensure the privacy and security of the text to be converted, the above-mentioned text to be converted can also be stored in a node of a block chain.

[0140] Based on the above-mentioned speech synthesis device 300, by performing phoneme conversion and context embedding on a text data set, corresponding phoneme sequences and text context vectors are obtained, NANSY features of a reference audio sample are extracted, the phoneme sequences and text context vectors of the input text are combined, a phoneme-level latent diffusion model and a phoneme-level variational auto-encoding model are used to generate target speech, and reference conditions and text conditions are ingeniously fused, so that the text content is accurately expressed while the speaker style is preserved, and then natural and fluent speech with rich expressiveness can be generated, while good timbre similarity and content accuracy are maintained, the speech synthesis quality is improved, and the balance between speaker fidelity and text intelligibility is significantly improved.

[0141] In some optional implementations of the embodiment, the conversion embedding module 303 includes:

[0142] The phoneme conversion submodule is configured to input the text data set into the phoneme conversion layer, map the text to phonemes by using a grapheme-to-phoneme conversion model, and obtain phoneme sequences.

[0143] The embedding submodule is configured to input the text data set into the context embedding layer, and perform context embedding vector representation on the text data set by using a pre-trained XLM-RoBERTa model to obtain text context vectors.

[0144] By converting text into phonemes and embedding its context, it helps to capture the phonetic features in the text and enhance the expressiveness of speech synthesis. It also helps to more accurately understand the meaning and contextual relationships of the text, improve the semantic accuracy of speech synthesis, and thus enhance the naturalness and fluency of speech synthesis.

[0145] In this embodiment, the XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer, and the embedding submodule includes:

[0146] An embedding unit, configured to perform vector conversion on text samples in the text dataset through the embedding layer to obtain a text embedding vector;

[0147] The encoding unit is used to input the text embedding vector into the Transformer encoder layer for encoding and conversion, and output a text context vector.

[0148] Contextual embedding through the XLM-RoBERTa model can convert the input text into a high-dimensional vector representation, thereby more accurately capturing the semantic features of the text and helping to enhance the comprehensibility of the text.

[0149] In some optional implementations, the phoneme-level latent diffusion model includes a phoneme encoder, a context encoder, a retrieval encoder, a conditioner, and a dual classifier-free latent diffusion encoder, and the encoding module 305 includes:

[0150] A phoneme encoding submodule, configured to encode the phoneme sequence using the phoneme encoder to obtain a phoneme feature vector;

[0151] A context encoding submodule, configured to input the phoneme feature vector and the text context vector into the context encoder for encoding to obtain a text-phoneme joint feature vector;

[0152] A retrieval encoding submodule, configured to encode the reference speech feature by using the retrieval encoder to obtain a speech style feature vector;

[0153] A conditional encoding submodule, configured to input the text phoneme joint feature vector and the speech style feature vector into the conditioner, fuse the reference condition and the text condition for encoding, and obtain a conditional feature vector;

[0154] The latent coding submodule is used to perform weight calculation on the conditional feature vector through the latent diffusion encoder to generate phoneme-level prior latent features.

[0155] The phoneme-level latent diffusion model is used to fuse reference conditions and text conditions to generate phoneme-level prior latent features, enhancing speaker fidelity and text comprehensibility. The retrieval encoder is used to capture the speaker's style in the reference speech features, achieving zero-sample speech synthesis capabilities, greatly reducing computing requirements, and improving model training efficiency.

[0156] In some optional implementations of this embodiment, the conditional encoding submodule is further configured to:

[0157] Inputting the text-phoneme joint feature vector and the speech style feature vector into the conditioner, wherein the conditioner includes a reference conditioner and a text conditioner;

[0158] fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner to perform style feature encoding to obtain a style condition feature vector;

[0159] Performing text feature encoding on the text-phoneme joint feature vector by the text conditioner to obtain a text condition feature vector;

[0160] The style conditional feature vector and the text conditional feature vector are output as conditional feature vectors.

[0161] By performing conditional encoding separately for the reference conditioner and the text conditioner, the conditions for different aspects of speech synthesis are decoupled and separated, and the conditions related to the reference speech and input text can be independently controlled, which enables extraordinary control over speaker fidelity and text comprehensibility, thereby increasing the flexibility of adjusting speaker fidelity or text comprehensibility.

[0162] In some optional implementations, the phoneme-level variational autoencoder model includes a latent feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder, and the decoding module 306 is further configured to:

[0163] Inputting the phoneme-level prior latent features into the latent feature decoder, the duration predictor, the phoneme prosody decoder and the upsampler respectively;

[0164] Performing phoneme-level duration prediction on the phoneme-level priori latent feature by the duration predictor to obtain an audio prediction duration, and inputting the audio prediction duration into the latent feature decoder;

[0165] Decoding the phoneme-level priori latent feature and the audio prediction duration by the latent feature decoder to obtain a predicted phoneme timing decoding feature;

[0166] The predicted phoneme prosody feature is obtained by fusing the predicted phoneme temporal decoding feature and the phoneme-level priori latent feature through the phoneme prosody decoder for decoding;

[0167] upsample the predicted phoneme prosody features and the phoneme-level prior latent features through the up-sampler to obtain frame-level features;

[0168] input the frame-level features into the frame-level decoder to obtain reconstructed speech features.

[0169] The phoneme-level prior latent features are decoded by the phoneme-level variational auto-encoding model to obtain reconstructed speech features, which can utilize the information in the latent features to accurately reconstruct the speech features, and the reconstruction process can preserve important features of the reference audio signal, further improving the quality of speech synthesis and enhancing the accuracy of speech recognition.

[0170] In some optional implementations of the embodiment, the alignment module 307 is further configured to:

[0171] obtain a phoneme feature vector obtained by encoding the phoneme sequence by the phoneme encoder;

[0172] align the target speech feature and the phoneme feature vector by using a monotonic alignment search method through the aligner to obtain a phoneme alignment feature.

[0173] By aligning the target speech feature and the phoneme sequence, the accuracy of feature matching can be improved, the generalization ability of the model can be enhanced, and the interpretability and robustness of the model can be improved.

[0174] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the embodiment is shown in the figure.

[0175] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 which are connected to each other through a system bus. It should be noted that only the computer device 4 with the memory 41, the processor 42, and the network interface 43 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0176] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0177] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for a speech synthesis method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0178] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for executing the speech synthesis method.

[0179] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0180] By performing phoneme conversion and context embedding on the text dataset, the corresponding phoneme sequence and text context vector are obtained, and the NANSY features of the reference audio sample are extracted. Combined with the phoneme sequence and text context vector of the input text, the target speech is generated using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. It cleverly combines the reference conditions and text conditions, accurately expresses the text content while retaining the speaker's style, and can generate natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility.

[0181] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech synthesis method as described above.

[0182] By performing phoneme conversion and context embedding on the text dataset, the corresponding phoneme sequence and text context vector are obtained, and the NANSY features of the reference audio sample are extracted. Combined with the phoneme sequence and text context vector of the input text, the target speech is generated using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. It cleverly combines the reference conditions and text conditions, accurately expresses the text content while retaining the speaker's style, and can generate natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] Obviously, the above-described embodiments are only some embodiments but not all the embodiments of the present application, the preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features therein. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Acquire a text dataset and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample; Inputting the text dataset and the audio sample set into a pre-trained speech synthesis model for training, the speech synthesis model comprising a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model; Inputting the text data set into the phoneme conversion layer and the context embedding layer for processing respectively, and obtaining a phoneme sequence and a text context vector respectively; Extracting NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features; Inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature; Inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features; Extracting NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and aligning the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features; Upsampling the phoneme alignment features to the frame level to obtain target audio frame-level features; Iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model; The text to be converted is obtained, and the text to be converted is input into the conditionally controlled speech synthesis model to obtain the target synthesized speech.

2. The speech synthesis method according to claim 1, wherein: The step of inputting the text data set into the phoneme conversion layer and the context embedding layer for processing, and correspondingly obtaining a phoneme sequence and a text context vector comprises: Inputting the text data set into the phoneme conversion layer, mapping the text into phonemes by adopting a grapheme-to-phoneme conversion model to obtain a phoneme sequence; The text dataset is input into the context embedding layer, and the pre-trained XLM-RoBERTa model is used to perform context embedding vector representation on the text dataset to obtain a text context vector.

3. The speech synthesis method according to claim 2, wherein: The XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer. The step of using the pre-trained XLM-RoBERTa model to perform context embedding vector representation on the text dataset to obtain the text context vector includes: Performing vector conversion on text samples in the text dataset through the embedding layer to obtain a text embedding vector; The text embedding vector is input into the Transformer encoder layer for encoding and transformation, and a text context vector is output.

4. The speech synthesis method according to claim 1, wherein: The phoneme-level latent diffusion model includes a phoneme encoder, a context encoder, a retrieval encoder, a conditioner, and a dual classifier-free latent diffusion encoder. The step of inputting the phoneme sequence, the text context vector, and the reference speech feature into the phoneme-level latent diffusion model and fusing the reference condition and the text condition to generate a phoneme-level priori latent feature includes: Encoding the phoneme sequence by the phoneme encoder to obtain a phoneme feature vector; Inputting the phoneme feature vector and the text context vector into the context encoder for encoding to obtain a text phoneme joint feature vector; Encoding the reference speech feature by the retrieval encoder to obtain a speech style feature vector; Inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining a conditional feature vector; The conditional feature vector is weighted by the latent diffusion encoder to generate a phoneme-level priori latent feature.

5. The speech synthesis method according to claim 4, characterized in that The step of inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining the conditional feature vector comprises: Inputting the text-phoneme joint feature vector and the speech style feature vector into the conditioner, wherein the conditioner includes a reference conditioner and a text conditioner; fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner to perform style feature encoding to obtain a style condition feature vector; Performing text feature encoding on the text-phoneme joint feature vector by the text conditioner to obtain a text condition feature vector; The style conditional feature vector and the text conditional feature vector are output as conditional feature vectors.

6. The speech synthesis method according to claim 1, wherein: The phoneme-level variational autoencoder model includes a latent feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder. The step of inputting the phoneme-level prior latent feature into the phoneme-level variational autoencoder model for decoding to obtain the reconstructed speech feature includes: Inputting the phoneme-level prior latent features into the latent feature decoder, the duration predictor, the phoneme prosody decoder and the upsampler respectively; Performing phoneme-level duration prediction on the phoneme-level priori latent feature by the duration predictor to obtain an audio prediction duration, and inputting the audio prediction duration into the latent feature decoder; Decoding the phoneme-level priori latent feature and the audio prediction duration by the latent feature decoder to obtain a predicted phoneme timing decoding feature; The predicted phoneme prosody feature is obtained by fusing the predicted phoneme temporal decoding feature and the phoneme-level priori latent feature through the phoneme prosody decoder for decoding; fusing the predicted phoneme prosody feature and the phoneme-level priori latent feature through the upsampler to perform upsampling to obtain a frame-level feature; The frame-level features are input into the frame-level decoder for decoding to obtain reconstructed speech features.

7. The speech synthesis method according to claim 4, characterized in that: The step of using a pre-trained aligner to align the target speech feature with the phoneme sequence to obtain a phoneme alignment feature includes: Obtaining a phoneme feature vector obtained by encoding the phoneme sequence by the phoneme encoder; The aligner uses a monotone alignment search method to align the target speech feature with the phoneme feature vector to obtain a phoneme alignment feature.

8. A speech synthesis device, characterized in that: include: An acquisition module, configured to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample; An input module, configured to input the text dataset and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model; A conversion and embedding module, configured to input the text dataset into the phoneme conversion layer and the context embedding layer for processing, thereby obtaining a phoneme sequence and a text context vector respectively; A feature extraction module, configured to extract NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features; an encoding module, configured to input the phoneme sequence, the text context vector, and the reference speech feature into the phoneme-level latent diffusion model, and fuse the reference condition and the text condition to generate a phoneme-level priori latent feature; A decoding module, configured to input the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features; An alignment module is configured to extract NANSY features of the target audio sample using the NANSY synthesizer to obtain target speech features, and align the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features; An upsampling module, configured to upsample the phoneme alignment features to a frame level to obtain target audio frame-level features; An iteration module, configured to iteratively train the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model; The speech synthesis module is used to obtain the text to be converted, input the text to be converted into the conditionally controlled speech synthesis model, and obtain the target synthesized speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method according to any one of claims 1 to 7.

Images