Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, medical treatment and finance, and relates to a speech synthesis method, which comprises the following steps: performing phoneme conversion and context embedding on a text data set, obtaining a phoneme sequence and a text context vector, extracting reference speech features of a reference audio sample, inputting a phoneme-level potential diffusion model, and generating phoneme-level prior potential features; inputting the phoneme-level prior potential features into a phoneme-level variational self-encoding model to obtain reconstructed speech features; extracting a target voice feature of the target audio sample; performing alignment and up-sampling on the target voice feature and a phoneme sequence to obtain a target audio frame-level feature; and training based on the reconstructed speech features and the target audio frame-level features to obtain a conditional control speech synthesis model, and synthesizing the to-be-converted text speech. The invention further provides a speech synthesis device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted text can be stored in a block chain. The speaker fidelity and the text understandability are improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, financial technology and digital medical technology, and in particular to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] In the fields of medicine and finance, accurate information transmission is crucial. As an effective means of information transmission, text-to-speech (TTS) technology has made significant technological progress in recent years. The original intention of the design of the TTS model is to accurately replicate the diversity of human speech, including the unique identity of the speaker and the subtle differences in language, which is of great significance for improving the authenticity and affinity of information transmission. Especially in scenarios such as medical diagnosis reports and financial data interpretation, converting text into natural and fluent speech through TTS technology can significantly improve the accessibility and comprehension of information.

[0003] However, although TTS technology has made great progress in imitating the richness of human speech, how to achieve the best balance between speaker fidelity and text comprehensibility in the face of diverse control requirements is still a major challenge facing current technology. The characteristics of human speech are extremely complex and varied, including unique speaker identity, different voice rhythms, intonations, and language styles, all of which place extremely high demands on the synthesis capabilities of TTS models.

[0004] An ideal TTS model should be able to capture the essential characteristics of the speaker, such as timbre, speaking style, accent, and emotion, to achieve high-fidelity speech synthesis. At the same time, the model must ensure that the synthesized speech is clear and easy to understand, and maintain strong text comprehensibility to meet the basic needs of information transmission. However, in practical applications, there is often a certain conflict between speaker fidelity and text comprehensibility.

[0005] To date, most TTS research has relied on indicators such as speaker similarity, naturalness MOS (Mean Opinion Score), and word error rate (WER) to evaluate speaker fidelity and text comprehensibility. However, there is still relatively little research on how to independently control each element when the two conflict. Therefore, how to ensure speech clarity and text comprehensibility while maintaining high speaker fidelity has become a key issue that needs to be urgently addressed in the current TTS technology field. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, computer device and storage medium to solve the technical problem in the prior art that it is difficult to maintain a perfect balance between speaker fidelity and text comprehensibility.

[0007] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solution:

[0008] Acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0009] Inputting the text dataset and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0010] Inputting the text data set into the phoneme conversion layer and the context embedding layer for processing respectively, and obtaining a phoneme sequence and a text context vector correspondingly;

[0011] Extracting NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0012] Inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature;

[0013] Inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features;

[0014] Extracting NANSY features of the target audio sample by the NANSY synthesizer to obtain target speech features, and aligning the target speech features with the phoneme sequence by a pre-trained aligner to obtain phoneme alignment features;

[0015] Upsampling the phoneme alignment features to the frame level to obtain target audio frame level features;

[0016] Iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model;

[0017] The text to be converted is obtained, and the text to be converted is input into the conditionally controlled speech synthesis model to obtain the target synthesized speech.

[0018] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device, which adopts the following technical solution:

[0019] An acquisition module, used to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0020] An input module, used for inputting the text data set and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0021] A conversion embedding module, used to input the text data set into the phoneme conversion layer and the context embedding layer for processing, and obtain a phoneme sequence and a text context vector respectively;

[0022] A feature extraction module, used to extract NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0023] An encoding module, used for inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature;

[0024] A decoding module, used for inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features;

[0025] An alignment module, configured to extract NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and align the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features;

[0026] An upsampling module, used for upsampling the phoneme alignment features to a frame level to obtain target audio frame level features;

[0027] An iteration module, used for iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model;

[0028] The speech synthesis module is used to obtain the text to be converted, input the text to be converted into the conditionally controlled speech synthesis model, and obtain the target synthesized speech.

[0029] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0030] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method described above when executing the computer-readable instructions.

[0031] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0032] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the speech synthesis method described above are implemented.

[0033] Compared with the prior art, this application has the following beneficial effects:

[0034] The present application provides a speech synthesis method, which obtains the corresponding phoneme sequence and text context vector by performing phoneme conversion and context embedding on a text data set, extracts the NANSY features of a reference audio sample, combines the phoneme sequence and text context vector of an input text, and generates a target speech using a phoneme-level latent diffusion model and a phoneme-level variational autoencoder model, ingeniously combines reference conditions and text conditions, accurately expresses the text content while retaining the speaker's style, and can generate natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, thereby improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0037] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;

[0038] Figure 3 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present application;

[0039] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0044] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0046] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0047] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the speech synthesis device is generally arranged in the server / terminal device.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0049] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application, comprising the following steps:

[0050] Step S201: Acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample.

[0051] Collect audio sample sets. Audio sample sets can be collected through professional recording studio equipment or through public data platforms. Preprocess the collected raw audio sample data, including removing noise, adjusting audio quality, etc., to obtain the preprocessed audio sample set, obtain the text data set corresponding to the audio sample set, and annotate the audio sample set, match the corresponding text with the corresponding audio clip, so as to establish a mapping relationship between text and audio, and provide supervised learning data for subsequent model training.

[0052] In this embodiment, the electronic device (eg, Figure 1The server / terminal device shown in the figure) can receive the acquired text data set and audio sample set through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0053] The audio sample set includes a large number of reference audio samples and target audio samples, that is, a reference audio sample subset and a target audio sample subset, wherein the reference audio sample subset is a reference speech sampled from the target audio sample subset. The audio sample set may be speech samples containing different accents and dialects.

[0054] For example, in the financial field, the audio sample set can be audio data containing financial professional terms, financial market data, stock trading information, financial reports, etc., or it can be recordings of conversations between financial customer service and customers; in the medical field, the audio sample set can be audio data containing medical diagnosis, treatment plans, drug names, disease names, etc., or it can be audio data containing medical guidance information such as drug instructions, health examination instructions, disease prevention suggestions, etc., or it can be recordings of conversations between doctors and patients of different age groups and genders during the diagnosis and treatment process.

[0055] Step S202: input the text data set and the audio sample set into a pre-trained speech synthesis model for training. The speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model.

[0056] In this embodiment, the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model and a phoneme-level variational autoencoder model. The speech synthesis model is trained through two stages, namely, VAE (variational autoencoder model) training and LDM (latent diffusion model) training. The input text is converted into a phoneme sequence at the phoneme level through the phoneme conversion layer, and the input text is converted into a text context vector through the context embedding layer. VAE training reconstructs the NANSY features of the reference audio sample based on the given target audio sample and phoneme sequence, and LDM training generates VAE latent features based on the given input text and reference audio sample.

[0057] In this embodiment, before the audio sample set is input into the speech synthesis model, it is first preprocessed, including denoising, normalization and other operations. Specifically, a wavelet denoising algorithm is used to perform denoising on the speech signal in the audio sample set to remove background noise interference; then the speech signal in the denoised audio sample set is normalized to map the amplitude of the speech signal to the interval [-1,1] to obtain a standardized audio sample set.

[0058] Step S203, input the text data set into the phoneme conversion layer and the context embedding layer for processing, and obtain the phoneme sequence and text context vector accordingly.

[0059] Specifically, the text dataset is input into the phoneme conversion layer, and the text is mapped into phonemes by adopting the grapheme-to-phoneme conversion model to obtain a phoneme sequence; the text dataset is input into the context embedding layer, and the pre-trained XLM-RoBERTa model is used to represent the text dataset with a context embedding vector to obtain a text context vector.

[0060] Among them, the phoneme conversion layer can convert text into corresponding pronunciation representations. Phonemes are symbolic representations of speech pronunciation. Converting text into phonemes helps the model accurately learn the phoneme mapping of text. Specifically, the phoneme conversion layer predicts the corresponding phoneme sequence by capturing the local and global patterns of the input sequence. The context embedding layer uses the XLM-RoBERTa model, which can convert the input text dataset into a vector representation containing the context information of each word or subword in the text.

[0061] By converting text into phonemes and embedding the context, it helps to capture the speech features in the text and enhance the expressiveness of speech synthesis. It also helps to more accurately understand the meaning and context of the text and improve the semantic accuracy of speech synthesis, thereby enhancing the naturalness and fluency of speech synthesis.

[0062] In some optional implementations of the present embodiment, the glyph-to-phoneme conversion model is a G2P model. Specifically, the input text data set is preprocessed, including operations such as word segmentation and part-of-speech tagging, to extract text features such as keywords and phrases; according to preset rules and templates, the extracted text features are converted into parameterized representations to obtain text feature parameters; the text feature parameters are input into a pre-trained G2P model, and the phoneme representation vector corresponding to each text feature parameter is obtained through forward propagation calculation; a beam search-based decoding algorithm is used to decode the phoneme representation vector output by the G2P model, and convert it into a phoneme symbol sequence to obtain a complete phoneme sequence representation; the generated phoneme sequence is post-processed, such as phoneme merging, deletion, and other operations, to optimize the fluency and naturalness of the phoneme sequence; and the optimized phoneme sequence is used for subsequent operations.

[0063] In some optional implementations of this embodiment, the XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer; the step of using the pre-trained XLM-RoBERTa model to represent the context embedding vector of the text dataset to obtain the text context vector includes:

[0064] The text samples in the text dataset are converted into vectors through the embedding layer to obtain the text embedding vector;

[0065] The text embedding vector is input into the Transformer encoder layer for encoding and transformation, and the text context vector is output.

[0066] In this embodiment, the text samples in the text data set are respectively embedded in words (WordEmbedding), positions (Position Embedding) and segments (Segment Embedding) through the embedding layer to obtain the corresponding word embedding vectors, position embedding vectors and segment embedding vectors, and the word embedding vectors, position embedding vectors and segment embedding vectors are concatenated to obtain the text embedding vector. The Transformer encoder layer is composed of a plurality of encoders stacked, each of which includes a plurality of self-attention mechanisms and a feedforward neural network for encoding and transforming the input text embedding vector, wherein each layer of encoder receives the output of the previous layer of encoder as input, and outputs a series of new vector representations, namely, the text context vector.

[0067] In this embodiment, the XLM-RoBERTa model can also learn to switch between different languages.

[0068] By using the XLM-RoBERTa model for context embedding, the input text can be converted into a high-dimensional vector representation, thereby more accurately capturing the semantic features of the text and helping to enhance the comprehensibility of the text.

[0069] Step S204, using a NANSY synthesizer to extract NANSY features of the reference audio sample to obtain reference speech features.

[0070] In this embodiment, a pre-trained NANSY synthesizer is used to extract NANSY features of the reference audio sample, and the NANSY features include language features, fundamental frequency (f0) features, periodic amplitude features, and non-periodic amplitude features.

[0071] Among them, the NANSY synthesizer mainly consists of two parts: an analysis module and a synthesis module. The analysis module is used for speech feature extraction and can extract key feature information from the input speech signal reference audio sample; the synthesis module is used for speech synthesis. Based on the features extracted by the analysis module, the synthesis module can generate high-quality speech output.

[0072] Step S205 , inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, and fusing the reference condition and the text condition to generate a phoneme-level priori latent feature.

[0073] In this embodiment, the phoneme-level latent diffusion model is a pre-trained model, including a phoneme encoder, a context encoder, a retrieval encoder, a conditioner, and a dual classifier-free guidance (CFG) latent diffusion encoder. The phoneme-level latent diffusion model uses dual classifier-free guidance to enhance speaker fidelity and text comprehensibility.

[0074] Specifically, the phoneme sequence is encoded through a phoneme encoder to obtain a phoneme feature vector; the phoneme feature vector and the text context vector are input into a context encoder for encoding to obtain a text phoneme joint feature vector; the reference speech features are encoded through a retrieval encoder to obtain a speech style feature vector; the text phoneme joint feature vector and the speech style feature vector are input into a conditioner, the reference condition and the text condition are fused for encoding to obtain a conditional feature vector; the conditional feature vector is weighted by a latent diffusion encoder to generate a phoneme-level prior latent feature.

[0075] In this embodiment, the conditioner and the dual classifier-free latent diffusion encoder are two main components. The conditioner generates phoneme-level conditional information based on the reference speaker and the input text. The conditioner shares the input from the context encoder. The context encoder is a Transformer encoder structure using a cross-attention mechanism to simulate the context-aware features obtained from the output of the phoneme encoder and the text context vector.

[0076] In some optional implementations, to reduce the mismatch between training and inference, noise is added to the reference audio sample before NANSY feature extraction, and the sample is cut into random lengths to obtain a reference audio sample with noise, and NANSY features are extracted from the reference audio sample with noise.

[0077] In this embodiment, the reference speech features are input into the retrieval encoder, and the weights are calculated through the cross-attention mechanism of the retrieval encoder. The query vector of the cross-attention mechanism of the retrieval encoder is a fixed-length token, which can be called a prototype. Therefore, the output of the retrieval encoder is also a fixed-length token, which encapsulates the speaker style of the reference audio sample.

[0078] The conditioner uses the speaker style and input text to encode speaker-related conditions and text content-related conditions to obtain a conditional feature vector of speaker-aware phoneme-level conditions. Based on the conditional feature vector, the latent features are obtained by adjusting the highly controllable CFG weights through the latent diffusion encoder.

[0079] The phoneme-level latent diffusion model is used to fuse reference conditions and text conditions to generate phoneme-level prior latent features, thereby enhancing speaker fidelity and text comprehensibility. The retrieval encoder is used to capture the speaker's style in the reference speech features, achieving zero-sample speech synthesis capability, greatly reducing computing requirements and improving model training efficiency. The CFG weights are adjusted through the latent diffusion encoder to achieve high controllability of the dual CFG, thereby improving the balance between speaker fidelity and text comprehensibility.

[0080] In some optional implementations of this embodiment, the step of inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining the conditional feature vector includes:

[0081] Inputting the text phoneme joint feature vector and the speech style feature vector into a conditioner, wherein the conditioner includes a reference conditioner and a text conditioner;

[0082] By fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner, the style feature is encoded to obtain the style condition feature vector;

[0083] Performing text feature encoding on the text-phoneme joint feature vector through a text conditioner to obtain a text condition feature vector;

[0084] The style conditional feature vector and the text conditional feature vector are output as conditional feature vectors.

[0085] The goal of the reference conditioner is to generate speaker-aware phoneme-level conditions by using speaker tags, i.e., the speech style feature vector output by the retrieval encoder, as the value of the cross-attention, and using a prototype of the same length as the retriever encoder to encode speaker-related conditions, i.e., obtain the style condition feature vector. The text conditioner relies only on text input, i.e., the text-phoneme joint feature vector, and obtains the text condition feature vector by encoding conditions related to the text content.

[0086] By performing conditional encoding separately for the reference conditioner and the text conditioner, the conditions for different aspects of speech synthesis are decoupled and separated, and the conditions related to the reference speech and the input text can be independently controlled, which enables extraordinary control over speaker fidelity and text comprehensibility, thereby increasing the flexibility of adjusting speaker fidelity or text comprehensibility.

[0087] In this embodiment, the architecture of the latent diffusion encoder is based on the Transformer encoder structure. Specifically, the latent diffusion encoder includes an embedding layer, a plurality of stacked encoders, and two MLP layers, and a conditional mechanism is simply added after the two MLP layers.

[0088] The style-conditioned feature vector and the text-conditioned feature vector are input into the latent diffusion encoder. First, the CFG (Control Free Generation) weight of the style-conditioned feature vector is adjusted to adjust the similarity with the speaker, and the CFG weight of the text-conditioned feature vector is adjusted to control the comprehensibility of the text details.

[0089] In some optional implementations, the phoneme encoder, context encoder, retrieval encoder, reference conditioner, text conditioner and latent diffusion encoder all adopt the Transformer encoder architecture. The number of encoders in the Transformer encoder architecture is set according to actual needs and is not limited here.

[0090] In this embodiment, the phoneme-level latent diffusion model is trained using L1 loss, and the formula is as follows:

[0091]

[0092] In the formula, ∈ represents the added noise, ∈ θ represents a diffusion model with parameters, μ is the mean estimated by the phoneme-level variational autoencoder model, t represents the time step, and Represents the noise coefficient corresponding to time t. In this application, the output c of the text conditioner is implemented text and the output c of the reference conditioner spk , to use the CFG weights at inference time.

[0093] Specifically, suppose that c is discarded with probability 5% text , by discarding c with a probability of 10% spk , and an additional 10% dropout is applied to both to promote the frequency of null conditional scenes. Training uses discrete integer diffusion time steps and noise schedules. t is uniformly sampled from [1, T], where T = 200. Following the approach of previous diffusion models, a linear variance schedule is used, defined as β i =β1+(β T-β1)(i-1) / (T-1), set β1=0.0001 and β T =0.03. Noise coefficient The calculation formula is

[0094] In this embodiment, the latent diffusion encoder adopts a dual CFG, which can be expressed as follows:

[0095]

[0096] in, represents the noise without classifier guidance, z t is the latent feature at time step t, defined as While Ф represents the zero tensor corresponding to the empty conditional state, ω spk Indicates c spk The CFG weight, ω text Indicates c text The CFG weight of .

[0097] Through the phoneme-level latent diffusion model, speech synthesis is more finely manipulated, directly addressing the problem of balancing text and speaker similarity and enhancing the model's comprehensibility and controllability.

[0098] Step S206: input the phoneme-level priori latent features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features.

[0099] The phoneme-level variational autoencoder model (VAE) processes the phoneme-level prior latent features output by the phoneme-level latent diffusion model to reconstruct the NANSY features of the reference audio samples.

[0100] In this embodiment, the phoneme-level variational autoencoder model includes a latent feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder.

[0101] Specifically, the phoneme-level prior latent features are respectively input into the latent feature decoder, the duration predictor, the phoneme prosody decoder and the upsampler; the phoneme-level prior latent features are predicted at the phoneme-level duration by the duration predictor to obtain the audio prediction duration, and the audio prediction duration is input into the latent feature decoder; the phoneme-level prior latent features and the audio prediction duration are decoded by the latent feature decoder to obtain the predicted phoneme temporal decoding features; the predicted phoneme temporal decoding features and the phoneme-level prior latent features are fused and decoded by the phoneme prosody decoder to obtain the predicted phoneme prosody features; the predicted phoneme prosody features and the phoneme-level prior latent features are fused and upsampled by the upsampler to obtain the frame-level features; the frame-level features are input into the frame-level decoder for decoding to obtain the reconstructed speech features.

[0102] In this embodiment, the decoding process starts with decoding the phoneme-level prior latent features through a latent feature decoder, and the phoneme-level prior latent features are input in parallel to the duration predictor, the phoneme prosody decoder and the upsampler. The duration predictor is responsible for estimating the duration of the phoneme level, and the phoneme prosody decoder is responsible for estimating the prosodic features of the phoneme level. The phoneme fundamental frequency (f0) can be obtained based on the phoneme prosodic features.

[0103] The architecture of the upsampler is almost the same as that in Parallel Tacotron 2, but does not include the channel dimension. The frame-level features upsampled by the upsampler are input to the frame-level decoder for decoding to reconstruct the NANSY features of the reference audio samples and obtain the reconstructed speech features.

[0104] The phoneme-level prior latent features are decoded through the phoneme-level variational autoencoder model to obtain reconstructed speech features. The information in the latent features can be used to accurately reconstruct the speech features, and the reconstruction process can retain the important features of the reference audio signal, further improving the quality of speech synthesis and enhancing the accuracy of speech recognition.

[0105] Step S207, extracting NANSY features of the target audio sample through a NANSY synthesizer to obtain target speech features, and using a pre-trained aligner to align the target speech features with the phoneme sequence to obtain phoneme alignment features.

[0106] During the training of the speech synthesis model, the target audio sample is introduced to guide the reconstruction of the NANSY features of the reference audio sample. The target audio sample corresponding to the reference audio sample is extracted by the pre-trained NANSY synthesizer to obtain the target speech feature. The phoneme sequence and the target speech feature are input into the posterior encoder introduced for model training for encoding to obtain the phoneme-level posterior latent feature. The phoneme-level posterior latent feature is input into the latent feature decoder. The target speech feature is aligned with the phoneme sequence using the pre-trained aligner to obtain the phoneme alignment feature.

[0107] In this embodiment, the step of using a pre-trained aligner to align the target speech feature with the phoneme sequence to obtain the phoneme alignment feature includes:

[0108] Obtain a phoneme feature vector obtained by encoding the phoneme sequence by a phoneme encoder;

[0109] Through the aligner, the target speech feature is aligned with the phoneme feature vector using a monotone alignment search method to obtain the phoneme alignment feature.

[0110] Specifically, an alignment matrix of size M*N is created by the aligner, where M represents the length of the target speech feature vector and N represents the length of the phoneme sequence vector, and the alignment matrix is ​​initialized; a dynamic programming algorithm is used to traverse the alignment matrix, and the value of each element is calculated according to the monotone alignment search method; if the current target speech feature vector element matches the current phoneme sequence vector element, then the value of the current matrix element is the upper left element value, and 1 is added; otherwise, the value of the current matrix element is the maximum value of the left element value and the upper element value. Backtracking from the lower right corner element of the alignment matrix to obtain the best alignment path; if the value of the current matrix element is equal to the upper left element value, 1 is added, and the current target speech feature vector element is aligned with the current phoneme sequence vector element and moved to the upper left corner; otherwise, if the value of the current matrix element is equal to the upper element value, then move up; if the value of the current matrix element is equal to the left element value, then move left; according to the alignment path obtained by backtracking, the alignment relationship between the target speech feature and the phoneme sequence is determined, and the alignment information at the phoneme level, that is, the phoneme alignment feature, is obtained.

[0111] By aligning the target speech features with the phoneme sequence, the accuracy of feature matching can be improved, the generalization ability of the model can be enhanced, and the interpretability and robustness of the model can be improved.

[0112] Step S208, upsampling the phoneme alignment features to the frame level to obtain target audio frame level features.

[0113] Specifically, the phoneme alignment features are input into the duration predictor to obtain the actual duration of the audio; the actual duration of the audio is input into the latent feature decoder, and the phoneme-level posterior latent features and the actual duration of the audio are decoded by the latent feature decoder to obtain the target phoneme timing features; the target phoneme timing features and the phoneme-level posterior latent features are input into the phoneme prosody decoder for decoding to obtain the target phoneme prosody features; the target phoneme prosody features and the phoneme-level posterior latent features are fused and up-sampled by the upsampler to obtain the target audio frame-level features.

[0114] Step S209, iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until the iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model.

[0115] In this embodiment, according to the preset loss function, the loss value is calculated based on the reconstructed speech features and the target audio frame-level features, and the Adam optimizer is used to adjust the model parameters of the speech synthesis model according to the loss value. The adjusted model continues to be iteratively trained until the iteration stop condition is met, that is, the number of iterations reaches the preset number of times or the first loss value does not change significantly, and a conditionally controlled speech synthesis model that has completed training is obtained.

[0116] In some optional implementations, the preset loss function includes one or more combinations of NANSY feature reconstruction loss, phoneme-level fundamental frequency reconstruction loss, duration loss, KL divergence of latent variables, and adversarial loss. Among them, NANSY feature reconstruction loss represents the loss between the reconstructed speech feature and the target speech feature; the phoneme-level fundamental frequency reconstruction loss represents the loss between the predicted phoneme prosodic feature and the target phoneme prosodic feature; the duration loss represents the loss between the predicted audio duration and the actual audio duration. The adversarial loss is trained by adding a discriminator. Specifically, the discriminator consists of a simple convolutional network and is trained using least squares loss and feature matching loss.

[0117] When the preset loss function is a combination of multiple combinations, the comprehensive loss value is obtained by weighted summing up each loss as the training loss.

[0118] Step S210, obtaining the text to be converted, and inputting the text to be converted into a conditional control speech synthesis model to obtain the target synthesized speech.

[0119] In this embodiment, the acquired text input condition to be converted controls the speech synthesis model for inference to generate the target synthesized speech. During the inference process, the phoneme-level potential diffusion model is used to generate phoneme-level prior latent variables through iterative denoising, specifically, a fast sampling method is adopted, using a variance noise plan, such as [1e-4, 5e-4, 1e-3, 5e-3, 0.01, 0.02, 0.05, 0.2, 0.3, 0.5, 0.4, 0.3, 0.3, 0.2, 0.1, 0.1]; the generated phoneme-level prior latent variables are then processed by the phoneme-level variational autoencoder model, including upsampling to the NANSY frame level and estimating the NANSY features; finally, the speech waveform is synthesized by the NANSY synthesizer according to the estimated NANSY features to obtain the target synthesized speech.

[0120] It should be emphasized that in order to further ensure the privacy and security of the text to be converted, the above text to be converted can also be stored in a node of a blockchain.

[0121] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0122] This application obtains the corresponding phoneme sequence and text context vector by performing phoneme conversion and context embedding on the text dataset, and extracts the NANSY features of the reference audio sample. The target speech is generated by combining the phoneme sequence and text context vector of the input text using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. The reference condition and the text condition are cleverly integrated to accurately express the text content while retaining the speaker's style, thereby generating natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, improving the quality of speech synthesis, and significantly improving the balance between speaker fidelity and text comprehensibility.

[0123] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0124] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0126] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0127] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 3 As shown, the speech synthesis device 300 described in this embodiment includes: an acquisition module 301, an input module 302, a conversion embedding module 303, a feature extraction module 304, an encoding module 305, a decoding module 306, an alignment module 307, an upsampling module 308, an iteration module 309 and a speech synthesis module 310. Among them:

[0129] The acquisition module 301 is used to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample;

[0130] The input module 302 is used to input the text data set and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model;

[0131] The conversion embedding module 303 is used to input the text data set into the phoneme conversion layer and the context embedding layer for processing, and obtain a phoneme sequence and a text context vector respectively;

[0132] The feature extraction module 304 is used to extract the NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features;

[0133] The encoding module 305 is used to input the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level potential diffusion model, and fuse the reference condition and the text condition to generate a phoneme-level priori potential feature;

[0134] The decoding module 306 is used to input the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features;

[0135] The alignment module 307 is used to extract the NANSY features of the target audio sample through the NANSY synthesizer to obtain the target speech features, and align the target speech features with the phoneme sequence using a pre-trained aligner to obtain the phoneme alignment features;

[0136] The upsampling module 308 is used to upsample the phoneme alignment features to the frame level to obtain target audio frame level features;

[0137] The iteration module 309 is used to iteratively train the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met to obtain a final conditionally controlled speech synthesis model;

[0138] The speech synthesis module 310 is used to obtain the text to be converted, input the text to be converted into the conditionally controlled speech synthesis model, and obtain the target synthesized speech.

[0139] It should be emphasized that in order to further ensure the privacy and security of the text to be converted, the above text to be converted can also be stored in a node of a blockchain.

[0140] Based on the above-mentioned speech synthesis device 300, by performing phoneme conversion and context embedding on the text data set, the corresponding phoneme sequence and text context vector are obtained, and the NANSY features of the reference audio sample are extracted. The phoneme sequence and text context vector of the input text are combined, and the target speech is generated using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. The reference conditions and text conditions are cleverly integrated, and the text content is accurately expressed while retaining the speaker's style, thereby being able to generate natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, thereby improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility.

[0141] In some optional implementations of this embodiment, the conversion embedding module 303 includes:

[0142] A phoneme conversion submodule, used for inputting the text data set into the phoneme conversion layer, mapping the text into phonemes by adopting a grapheme-to-phoneme conversion model, and obtaining a phoneme sequence;

[0143] The embedding submodule is used to input the text data set into the context embedding layer, use the pre-trained XLM-RoBERTa model to perform context embedding vector representation on the text data set, and obtain a text context vector.

[0144] By converting text into phonemes and embedding the context, it helps to capture the speech features in the text and enhance the expressiveness of speech synthesis. It also helps to more accurately understand the meaning and context of the text and improve the semantic accuracy of speech synthesis, thereby enhancing the naturalness and fluency of speech synthesis.

[0145] In this embodiment, the XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer, and the embedding submodule includes:

[0146] An embedding unit, used for performing vector conversion on text samples in the text data set through the embedding layer to obtain a text embedding vector;

[0147] The encoding unit is used to input the text embedding vector into the Transformer encoder layer for encoding and conversion, and output a text context vector.

[0148] By using the XLM-RoBERTa model for context embedding, the input text can be converted into a high-dimensional vector representation, thereby more accurately capturing the semantic features of the text and helping to enhance the comprehensibility of the text.

[0149] In some optional implementations, the phoneme-level latent diffusion model includes a phoneme encoder, a context encoder, a retrieval encoder, a conditioner, and a dual classifier-free latent diffusion encoder, and the encoding module 305 includes:

[0150] A phoneme encoding submodule, used for encoding the phoneme sequence through the phoneme encoder to obtain a phoneme feature vector;

[0151] A context encoding submodule, used for inputting the phoneme feature vector and the text context vector into the context encoder for encoding to obtain a text-phoneme joint feature vector;

[0152] A retrieval encoding submodule, used for encoding the reference speech feature through the retrieval encoder to obtain a speech style feature vector;

[0153] A conditional encoding submodule, used for inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining a conditional feature vector;

[0154] The latent coding submodule is used to perform weight calculation on the conditional feature vector through the latent diffusion encoder to generate a phoneme-level priori latent feature.

[0155] The phoneme-level latent diffusion model is used to fuse reference conditions and text conditions to generate phoneme-level prior latent features, enhance speaker fidelity and text comprehensibility, and capture the speaker's style in the reference speech features through the retrieval encoder to achieve zero-sample speech synthesis capability, greatly reduce computing requirements, and improve model training efficiency.

[0156] In some optional implementations of this embodiment, the conditional encoding submodule is further used to:

[0157] Inputting the text-phoneme joint feature vector and the speech style feature vector into the conditioner, wherein the conditioner includes a reference conditioner and a text conditioner;

[0158] fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner to perform style feature encoding to obtain a style condition feature vector;

[0159] Performing text feature encoding on the text-phoneme joint feature vector by the text conditioner to obtain a text condition feature vector;

[0160] The style condition feature vector and the text condition feature vector are output as condition feature vectors.

[0161] By performing conditional encoding separately for the reference conditioner and the text conditioner, the conditions for different aspects of speech synthesis are decoupled and separated, and the conditions related to the reference speech and the input text can be independently controlled, which enables extraordinary control over speaker fidelity and text comprehensibility, thereby increasing the flexibility of adjusting speaker fidelity or text comprehensibility.

[0162] In some optional implementations, the phoneme-level variational autoencoder model includes a latent feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder, and the decoding module 306 is further used to:

[0163] Inputting the phoneme-level prior latent features into the latent feature decoder, the duration predictor, the phoneme prosody decoder and the upsampler respectively;

[0164] Performing phoneme-level duration prediction on the phoneme-level priori latent feature by the duration predictor to obtain an audio prediction duration, and inputting the audio prediction duration into the latent feature decoder;

[0165] Decoding the phoneme-level a priori latent feature and the audio prediction duration by the latent feature decoder to obtain a predicted phoneme timing decoding feature;

[0166] The predicted phoneme prosody feature is obtained by fusing the predicted phoneme temporal decoding feature and the phoneme-level priori latent feature through the phoneme prosody decoder for decoding;

[0167] The upsampler fuses the predicted phoneme prosody feature and the phoneme-level priori latent feature to perform upsampling to obtain a frame-level feature;

[0168] The frame-level features are input into the frame-level decoder for decoding to obtain reconstructed speech features.

[0169] The phoneme-level prior latent features are decoded through the phoneme-level variational autoencoder model to obtain reconstructed speech features. The information in the latent features can be used to accurately reconstruct the speech features, and the reconstruction process can retain the important features of the reference audio signal, further improving the quality of speech synthesis and enhancing the accuracy of speech recognition.

[0170] In some optional implementations of this embodiment, the alignment module 307 is further configured to:

[0171] Obtaining a phoneme feature vector obtained by encoding the phoneme sequence by the phoneme encoder;

[0172] The target speech feature is aligned with the phoneme feature vector by the aligner in a monotone alignment search manner to obtain a phoneme alignment feature.

[0173] By aligning the target speech features with the phoneme sequence, the accuracy of feature matching can be improved, the generalization ability of the model can be enhanced, and the interpretability and robustness of the model can be improved.

[0174] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0175] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0176] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0177] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the speech synthesis method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0178] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the speech synthesis method.

[0179] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0180] By performing phoneme conversion and context embedding on the text dataset, the corresponding phoneme sequence and text context vector are obtained, and the NANSY features of the reference audio sample are extracted. Combined with the phoneme sequence and text context vector of the input text, the target speech is generated using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. The reference condition and text condition are cleverly integrated to accurately express the text content while retaining the speaker's style, thereby generating natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility.

[0181] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech synthesis method as described above.

[0182] By performing phoneme conversion and context embedding on the text dataset, the corresponding phoneme sequence and text context vector are obtained, and the NANSY features of the reference audio sample are extracted. Combined with the phoneme sequence and text context vector of the input text, the target speech is generated using the phoneme-level latent diffusion model and the phoneme-level variational autoencoder model. The reference condition and text condition are cleverly integrated to accurately express the text content while retaining the speaker's style, thereby generating natural, fluent and expressive speech while maintaining good timbre similarity and content accuracy, improving the quality of speech synthesis and significantly improving the balance between speaker fidelity and text comprehensibility.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample; Inputting the text dataset and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model; Inputting the text data set into the phoneme conversion layer and the context embedding layer for processing respectively, and obtaining a phoneme sequence and a text context vector correspondingly; Extracting NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features; Inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature; Inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features; Extracting NANSY features of the target audio sample by the NANSY synthesizer to obtain target speech features, and aligning the target speech features with the phoneme sequence by a pre-trained aligner to obtain phoneme alignment features; Upsampling the phoneme alignment features to the frame level to obtain target audio frame level features; Iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model; The text to be converted is obtained, and the text to be converted is input into the conditionally controlled speech synthesis model to obtain the target synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that: The step of inputting the text data set into the phoneme conversion layer and the context embedding layer for processing respectively, and correspondingly obtaining the phoneme sequence and the text context vector comprises: Inputting the text data set into the phoneme conversion layer, mapping the text into phonemes by adopting a grapheme-to-phoneme conversion model, and obtaining a phoneme sequence; The text data set is input into the context embedding layer, and the pre-trained XLM-RoBERTa model is used to perform context embedding vector representation on the text data set to obtain a text context vector.

3. The speech synthesis method according to claim 2, characterized in that: The XLM-RoBERTa model includes an embedding layer and a Transformer encoder layer; the step of using the pre-trained XLM-RoBERTa model to perform context embedding vector representation on the text dataset to obtain a text context vector includes: Performing vector conversion on text samples in the text data set through the embedding layer to obtain a text embedding vector; The text embedding vector is input into the Transformer encoder layer for encoding and conversion, and a text context vector is output.

4. The speech synthesis method according to claim 1, characterized in that: The phoneme-level latent diffusion model comprises a phoneme encoder, a context encoder, a retrieval encoder, a conditioner and a dual classifier-free latent diffusion encoder, and the step of inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model and fusing the reference condition and the text condition to generate a phoneme-level priori latent feature comprises: Encoding the phoneme sequence by the phoneme encoder to obtain a phoneme feature vector; Inputting the phoneme feature vector and the text context vector into the context encoder for encoding to obtain a text phoneme joint feature vector; Encoding the reference speech feature by the retrieval encoder to obtain a speech style feature vector; Inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining a conditional feature vector; The conditional feature vector is weighted by the latent diffusion encoder to generate a phoneme-level priori latent feature.

5. The speech synthesis method according to claim 4, characterized in that: The step of inputting the text phoneme joint feature vector and the speech style feature vector into the conditioner, fusing the reference condition and the text condition for encoding, and obtaining the conditional feature vector comprises: Inputting the text-phoneme joint feature vector and the speech style feature vector into the conditioner, wherein the conditioner includes a reference conditioner and a text conditioner; fusing the text phoneme joint feature vector and the speech style feature vector through the reference conditioner to perform style feature encoding to obtain a style condition feature vector; Performing text feature encoding on the text-phoneme joint feature vector by the text conditioner to obtain a text condition feature vector; The style condition feature vector and the text condition feature vector are output as condition feature vectors.

6. The speech synthesis method according to claim 1, characterized in that: The phoneme-level variational autoencoder model includes a potential feature decoder, a duration predictor, a phoneme prosody decoder, an upsampler, and a frame-level decoder. The step of inputting the phoneme-level priori potential feature into the phoneme-level variational autoencoder model for decoding to obtain the reconstructed speech feature includes: Inputting the phoneme-level prior latent features into the latent feature decoder, the duration predictor, the phoneme prosody decoder and the upsampler respectively; Performing phoneme-level duration prediction on the phoneme-level priori latent feature by the duration predictor to obtain an audio prediction duration, and inputting the audio prediction duration into the latent feature decoder; Decoding the phoneme-level a priori latent feature and the audio prediction duration by the latent feature decoder to obtain a predicted phoneme timing decoding feature; The predicted phoneme prosody feature is obtained by fusing the predicted phoneme temporal decoding feature and the phoneme-level priori latent feature through the phoneme prosody decoder for decoding; The upsampler fuses the predicted phoneme prosody feature and the phoneme-level priori latent feature to perform upsampling to obtain a frame-level feature; The frame-level features are input into the frame-level decoder for decoding to obtain reconstructed speech features.

7. The speech synthesis method according to claim 4, characterized in that: The step of using a pre-trained aligner to align the target speech feature with the phoneme sequence to obtain a phoneme alignment feature comprises: Obtaining a phoneme feature vector obtained by encoding the phoneme sequence by the phoneme encoder; The target speech feature is aligned with the phoneme feature vector by the aligner in a monotone alignment search manner to obtain a phoneme alignment feature.

8. A speech synthesis device, characterized in that: include: An acquisition module, used to acquire a text data set and an audio sample set, wherein the audio sample set includes a reference audio sample and a target audio sample; An input module, used for inputting the text data set and the audio sample set into a pre-trained speech synthesis model for training, wherein the speech synthesis model includes a phoneme conversion layer, a context embedding layer, a NANSY synthesizer, a phoneme-level latent diffusion model, and a phoneme-level variational autoencoder model; A conversion embedding module, used to input the text data set into the phoneme conversion layer and the context embedding layer for processing, and obtain a phoneme sequence and a text context vector respectively; A feature extraction module, used to extract NANSY features of the reference audio sample using the NANSY synthesizer to obtain reference speech features; An encoding module, used for inputting the phoneme sequence, the text context vector and the reference speech feature into the phoneme-level latent diffusion model, fusing the reference condition and the text condition to generate a phoneme-level priori latent feature; A decoding module, used for inputting the phoneme-level prior potential features into the phoneme-level variational autoencoder model for decoding to obtain reconstructed speech features; An alignment module, configured to extract NANSY features of the target audio sample through the NANSY synthesizer to obtain target speech features, and align the target speech features with the phoneme sequence using a pre-trained aligner to obtain phoneme alignment features; An upsampling module, used for upsampling the phoneme alignment features to a frame level to obtain target audio frame level features; An iteration module, used for iteratively training the speech synthesis model based on the reconstructed speech features and the target audio frame-level features until an iteration stop condition is met, thereby obtaining a final conditionally controlled speech synthesis model; The speech synthesis module is used to obtain the text to be converted, input the text to be converted into the conditionally controlled speech synthesis model, and obtain the target synthesized speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images