Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belong to the technical field of artificial intelligence. The method comprises: obtaining original phoneme data; inputting the original phoneme data into a speech synthesis model; performing first spectrum prediction on the original phoneme data based on a first prediction network and a speaker corpus to obtain initial spectrum data and an initial speech embedding vector; performing filtering processing on the initial spectrum data based on the initial speech embedding vector to obtain candidate spectrum data; performing second spectrum prediction on the candidate spectrum data based on a second prediction network and the speaker corpus to obtain intermediate spectrum data and a target speech embedding vector; performing reconstruction processing on the intermediate spectrum data based on the target speech embedding vector to obtain target spectrum data; and performing vocoding on the target spectrum data based on a decoding network to obtain speech synthesis data. The embodiments of the present application can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, speech synthesis device, electronic device and storage medium. Background Technology

[0002] The training of neural network models used in current speech synthesis methods often requires a large amount of labeled data. However, in real-world scenarios, the cost of obtaining a large amount of high-quality labeled data is high, resulting in a shortage of labeled data for model training. This affects the training effect of the model and leads to poor accuracy of the speech synthesis data based on this model. Therefore, how to improve the accuracy of speech synthesis has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a speech synthesis method, speech synthesis device, electronic device, and storage medium, which aim to improve the accuracy of speech synthesis.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] Obtain raw phoneme data, wherein the raw phoneme data is text data;

[0006] The raw phoneme data is input into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network.

[0007] Based on the first prediction network and the pre-acquired corpus of speaking objects, the original phoneme data is subjected to a first spectrum prediction to obtain initial spectrum data and an initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking objects in the corpus of speaking objects.

[0008] The initial spectral data is filtered based on the initial speech embedding vector to obtain candidate spectral data.

[0009] Based on the second prediction network and the corpus of the speaking object, a second spectrum prediction is performed on the candidate spectrum data to obtain intermediate spectrum data and target speech embedding vector. The target speech embedding vector is used to characterize the speaking style features of the target speaking object in the corpus of the speaking object.

[0010] The intermediate spectral data is reconstructed based on the target speech embedding vector to obtain the target spectral data, which is a Mel spectrogram.

[0011] Based on the decoding network, the target spectrum data is vocode converted to obtain speech synthesis data.

[0012] In some embodiments, the first prediction network includes a first encoding layer and a first decoding layer. The first prediction network includes performing a first spectral prediction on the original phoneme data based on the first prediction network and a pre-acquired corpus of the speaking audience to obtain initial spectral data and an initial speech embedding vector, including:

[0013] Obtain the first duration parameter of the original phoneme data;

[0014] The original phoneme data is encoded using the first encoding layer and the first duration parameter to obtain the phoneme hidden layer representation vector.

[0015] Based on the corpus of the speaking object, feature extraction is performed on the phoneme hidden layer representation vector to obtain the initial speech embedding vector;

[0016] The initial spectral data is obtained by performing attention calculations on the phoneme hidden layer representation vectors through the first decoding layer.

[0017] In some embodiments, the first duration parameter for obtaining the raw phoneme data includes:

[0018] The original phoneme data is input into a preset time prediction model, which includes a convolutional layer, an activation layer, a normalization layer, and a linear layer.

[0019] The convolutional layer is used to extract features from the original phoneme data to obtain phoneme feature vectors;

[0020] The phoneme feature vector is mapped to a preset temporal space through the activation layer to obtain the initial predicted temporal features;

[0021] The initial prediction time features are standardized by the normalization layer to obtain the target prediction time features.

[0022] The target predicted time features are mapped to a preset data space through the linear layer to obtain the first duration parameter.

[0023] In some embodiments, the second prediction network includes a second encoding layer and a second decoding layer. The step of performing second spectral prediction on the candidate spectral data based on the second prediction network and the speech object corpus to obtain intermediate spectral data and a target speech embedding vector includes:

[0024] Obtain the second duration parameter of the candidate spectrum data;

[0025] The candidate spectral data is encoded using the second coding layer and the second duration parameter to obtain a speech hidden layer representation vector;

[0026] Based on the corpus of the speaking object, feature extraction is performed on the speech hidden layer representation vector to obtain the target speech embedding vector;

[0027] The intermediate spectral data is obtained by performing attention calculation on the speech hidden layer representation vector through the second decoding layer.

[0028] In some embodiments, the step of filtering the initial spectral data based on the initial speech embedding vector to obtain candidate spectral data includes:

[0029] Based on the initial speech embedding vector, feature recognition is performed on the initial spectral data to obtain background spectral features and key spectral features; wherein, the key spectral features include the speech content corresponding to the original phoneme data, and the background spectral features do not include the speech content corresponding to the original phoneme data;

[0030] The background spectral features in the initial spectral data are filtered to obtain the candidate spectral data.

[0031] In some embodiments, the process of reconstructing the intermediate spectral data based on the target speech embedding vector to obtain the target spectral data includes:

[0032] Feature extraction is performed on the target speech embedding vector to obtain the timbre features of the target speaker;

[0033] The timbre features are embedded into the intermediate spectral data to obtain the target spectral data, which contains the timbre characteristics of the target speaker.

[0034] In some embodiments, the decoding network includes an upsampling module and a residual module, and the step of performing vocode conversion on the target spectral data based on the decoding network to obtain speech synthesis data includes:

[0035] The target spectral data is upsampled using the upsampling module to obtain initial speech features.

[0036] The speech synthesis data is obtained by performing vocode conversion on the initial speech features using the residual module.

[0037] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0038] The data acquisition module is used to acquire raw phoneme data, which is text data;

[0039] An input module is used to input the raw phoneme data into a preset speech synthesis model, the speech synthesis model including a first prediction network, a second prediction network, and a decoding network;

[0040] The first spectrum prediction module is used to perform a first spectrum prediction on the original phoneme data based on the first prediction network and the pre-acquired speech object corpus to obtain initial spectrum data and an initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking object in the speech object corpus.

[0041] The filtering module is used to filter the initial spectrum data based on the initial speech embedding vector to obtain candidate spectrum data;

[0042] The second spectrum prediction module is used to perform a second spectrum prediction on the candidate spectrum data based on the second prediction network and the speech object corpus to obtain intermediate spectrum data and target speech embedding vector. The target speech embedding vector is used to characterize the speaking style features of the target speech object in the speech object corpus.

[0043] The reconstruction module is used to reconstruct the intermediate spectral data based on the target speech embedding vector to obtain target spectral data, which is a Mel spectrogram.

[0044] The vocode conversion module is used to perform vocode conversion on the target spectrum data based on the decoding network to obtain speech synthesis data.

[0045] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0046] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0047] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium proposed in this application acquire raw phoneme data, which is text data. The raw phoneme data is then input into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network. Based on the first prediction network and a pre-acquired corpus of the speaking audience, a first spectral prediction is performed on the raw phoneme data to obtain initial spectral data and an initial speech embedding vector. This enables preliminary prediction of the raw phoneme data and extraction of phoneme feature information. Based on the initial speech embedding vector, the initial spectral data is filtered to obtain candidate spectral data. Then, based on the second prediction network and the corpus of the speaking audience, a second spectral prediction is performed on the candidate spectral data to obtain intermediate spectral data and a target speech embedding vector. This enables further prediction of the raw phoneme data, further optimizing the quality of the spectral data and obtaining more accurate intermediate spectral data. Furthermore, the intermediate spectral data is reconstructed based on the target speech embedding vector to obtain the target spectral data, and the target spectral data is then converted into speech data based on the decoding network. This process can fuse the target speech embedding vector into the intermediate spectral data, so that the final target spectral data contains the speaking style characteristics of the target speaker, thereby improving the accuracy of speech synthesis. Attached Figure Description

[0048] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0049] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0050] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0051] Figure 4 yes Figure 1 The flowchart of step S104 in the process;

[0052] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0053] Figure 6 yes Figure 1 The flowchart of step S106 in the process;

[0054] Figure 7 yes Figure 1 The flowchart of step S107 in the process;

[0055] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0056] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] First, let's analyze some of the terms used in this application:

[0061] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0062] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0063] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0064] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0065] Phoneme: The smallest unit of speech based on the natural properties of speech. It is analyzed based on the articulation of a syllable, and one articulation constitutes one phoneme.

[0066] Activation function: This is a function that runs on the neurons of an artificial neural network and is responsible for mapping the input of the neuron to its output.

[0067] Encoder: Transforms an input sequence into a fixed-length vector.

[0068] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0069] Speech synthesis refers to the synthesis of intelligible and natural speech from text, also known as text-to-speech (TTS).

[0070] The training of neural network models used in current speech synthesis methods often requires a large amount of labeled data. However, in real-world scenarios, the cost of obtaining a large amount of high-quality labeled data is high, resulting in a shortage of labeled data for model training. This affects the training effect of the model and leads to poor accuracy of the speech synthesis data based on this model. Therefore, how to improve the accuracy of speech synthesis has become an urgent technical problem to be solved.

[0071] Based on this, embodiments of this application provide a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium, aiming to improve the accuracy of speech synthesis.

[0072] The speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0073] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0074] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0075] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0076] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0078] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0079] Step S101: Obtain the raw phoneme data, which is text data;

[0080] Step S102: Input the raw phoneme data into a preset speech synthesis model. The speech synthesis model includes a first prediction network, a second prediction network, and a decoding network.

[0081] Step S103: Based on the first prediction network and the pre-acquired speech object corpus, perform the first spectrum prediction on the original phoneme data to obtain initial spectrum data and initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking object in the speech object corpus.

[0082] Step S104: Filter the initial spectral data based on the initial speech embedding vector to obtain candidate spectral data;

[0083] Step S105: Perform second spectrum prediction on candidate spectrum data based on the second prediction network and the speech object corpus to obtain intermediate spectrum data and target speech embedding vector. The target speech embedding vector is used to characterize the speaking style features of the target speech object in the speech object corpus.

[0084] Step S106: Reconstruct the intermediate spectral data based on the target speech embedding vector to obtain the target spectral data, which is a Mel spectrogram;

[0085] Step S107: Based on the decoding network, the target spectrum data is vocode converted to obtain speech synthesis data.

[0086] Steps S101 to S107 of this embodiment involve acquiring raw phoneme data, which is text data; inputting the raw phoneme data into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network. Based on the first prediction network and a pre-acquired corpus of the speaking target, a first spectral prediction is performed on the raw phoneme data to obtain initial spectral data and an initial speech embedding vector, enabling preliminary prediction of the raw phoneme data and extraction of phoneme feature information. Based on the initial speech embedding vector, the initial spectral data is filtered to obtain candidate spectral data, and based on the second prediction network and the corpus of the speaking target, a second spectral prediction is performed on the candidate spectral data to obtain intermediate spectral data and a target speech embedding vector, enabling further prediction of the raw phoneme data, optimizing the quality of the spectral data, and obtaining more accurate intermediate spectral data. Furthermore, the intermediate spectral data is reconstructed based on the target speech embedding vector to obtain the target spectral data, and the target spectral data is then converted into speech data based on the decoding network. This process can fuse the target speech embedding vector into the intermediate spectral data, so that the final target spectral data contains the speaking style characteristics of the target speaker, thereby improving the accuracy of speech synthesis.

[0087] In step S101 of some embodiments, the raw text data can be obtained from a public dataset, or from an existing text database or online platform, without limitation. For example, the public dataset could be the THCHS30 dataset or the LJSpeech dataset, etc. The raw text data could include speech documents, music text materials, dubbing text materials, etc. Further, the raw text data is transformed according to a preset reference dictionary to obtain raw phoneme data. The preset reference dictionary can be a data dictionary containing multiple words. A word list can be constructed using the reference dictionary, with each line containing one word or character. This word list allows for data transformation of the text content in the raw text data, converting the corresponding words or characters into phoneme sequences, thereby forming raw phoneme data in text form. This method allows for convenient acquisition of raw phoneme data, improving data acquisition efficiency.

[0088] In step S102 of some embodiments, the original phoneme data can be input into a preset speech synthesis model through a preset script or other computer program. The speech synthesis model includes a first prediction network, a second prediction network, and a decoding network. The first prediction network is mainly used to perform preliminary spectrum prediction on the original phoneme data to obtain initial spectrum data and initial speech embedding vectors for each reference speaker. The second prediction network is mainly used to perform secondary spectrum prediction based on the initial spectrum data and initial speech embedding vectors to obtain more accurate intermediate spectrum data and target speech embedding vectors for the target speaker. The target speech embedding vectors, which characterize the speaking features of the target speaker, are then embedded into the intermediate spectrum data to obtain target spectrum data. The decoding network is mainly used to perform feature decoding on the target spectrum data to extract waveform-based speech synthesis data. This speech synthesis model can effectively embed the speech features of the target speaker into the speech synthesis data, enabling the speech synthesis data to contain the required prosodic features and speech content, thereby improving the accuracy of speech synthesis.

[0089] It should be noted that during the training of the speech synthesis model, a corpus of speaking subjects can be constructed based on the collected audio data used for speech synthesis. This process can specifically include:

[0090] a. Obtain the speech data of the reference speaker by means of web crawling, extracting from a preset database or downloading from a network platform. Divide the obtained speech data of the reference speaker into a single speech object corpus set and multiple speech object corpus sets. The single speech object corpus set contains only the speech data of a fixed speech object, while the multiple speech object corpus sets contain the speech data of multiple different speech objects.

[0091] b. Force alignment of all speech data at the phoneme level, and divide the speech data into speech segments of fixed length, i.e., all speech segments have the same number of frames;

[0092] c. Use a single speech object corpus that has been phoneme aligned to train the speech synthesis model. Use the trained speech synthesis model to synthesize speech from multiple speech object corpora to obtain synthesized data that is consistent with the transcription and phoneme duration of multiple speech object corpora. Incorporate these synthesized data into the same set and use this set as the final speech object corpus.

[0093] The aforementioned corpus of speech subjects can assist in model training, enabling the speech synthesis model to have better performance. It can also be used in subsequent speech synthesis processes to improve the accuracy of acquiring the speech characteristics of the target speech subject, so as to integrate the speech characteristics of the target speech subject into the generated speech synthesis data, thereby generating high-quality speech synthesis data.

[0094] Please see Figure 2 In some embodiments, the first prediction network includes a first encoding layer and a first decoding layer, and step S103 may include, but is not limited to, steps S201 to S205:

[0095] Step S201: Obtain the first duration parameter of the original phoneme data;

[0096] Step S202: Encode the original phoneme data using the first coding layer and the first duration parameter to obtain the phoneme hidden layer representation vector;

[0097] Step S203: Based on the speech object corpus, feature extraction is performed on the phoneme hidden layer representation vector to obtain the initial speech embedding vector;

[0098] Step S204: Attention calculation is performed on the phoneme hidden layer representation vector through the first decoding layer to obtain the initial spectrum data.

[0099] In step S201 of some embodiments, the original phoneme data can be input into a preset time prediction model. The time prediction model performs operations such as feature extraction and feature mapping on the original phoneme data to obtain the time features of the original phoneme data in the time domain. Then, the time features are standardized to obtain a first duration parameter. The first duration parameter can characterize the number of frames of each phoneme in the original phoneme data. The prediction duration characterized by the first duration parameter will affect the pronunciation length and prosodic characteristics of the generated speech synthesis data.

[0100] In step S202 of some embodiments, the original phoneme data is encoded by the first encoding layer and the first duration parameter to obtain the phoneme sequence features in the original phoneme data. The phoneme sequence in the text form of the original phoneme data is converted into a vector form to obtain the phoneme hidden layer representation vector.

[0101] In step S203 of some embodiments, the speech object corpus is traversed, and the phoneme hidden layer representation vector is compared with the speech data of reference speech objects in the speech object corpus. The similarity between the phoneme hidden layer representation vector and the candidate speech embedding vector of the speech data of each reference speech object is calculated, resulting in multiple feature similarity values. The candidate speech embedding vector with a feature similarity value greater than or equal to a preset feature threshold is used as the initial speech embedding vector of the reference speech object. The candidate speech embedding vector can be obtained by encoding the speech data of the reference speech object using a preset feature encoder. The reference speech objects include different speakers, different singers, or speakers of other identities, etc. The initial speech embedding vector can be used to represent the speaking style characteristics of the reference speech objects in the speech object corpus.

[0102] In step S204 of some embodiments, when performing attention calculation on the phoneme hidden layer representation vector through the first decoding layer, the query matrix, value matrix and key matrix of the phoneme hidden layer representation vector are first calculated, and then the query matrix, value matrix and key matrix are weighted by the softmax function to obtain the attention matrix corresponding to the phoneme hidden layer representation vector. Based on the attention matrix, all feature vectors in the phoneme hidden layer representation vector are weighted to obtain the initial spectrum data.

[0103] Through the above steps S201 to S204, a preliminary prediction of the original phoneme data can be achieved, phoneme feature information in the original phoneme data can be extracted, and initial spectrum data corresponding to the original phoneme data can be obtained.

[0104] Please see Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S305:

[0105] Step S301: Input the raw phoneme data into the preset time prediction model. The time prediction model includes a convolutional layer, an activation layer, a normalization layer, and a linear layer.

[0106] Step S302: Extract features from the original phoneme data through a convolutional layer to obtain phoneme feature vectors;

[0107] Step S303: The phoneme feature vector is mapped to a preset temporal space through the activation layer to obtain the initial predicted temporal features;

[0108] Step S304: The initial prediction time features are standardized by a normalization layer to obtain the target prediction time features;

[0109] Step S305: Map the target prediction time features to a preset data space through a linear layer to obtain the first duration parameter.

[0110] In step S301 of some embodiments, the raw phoneme data is input into a preset time prediction model through a preset script program or computer program. The time prediction model includes a convolutional layer, an activation layer, a normalization layer, and a linear layer.

[0111] In step S302 of some embodiments, the original phoneme data can be convolved and transposed by a convolutional layer to extract important phoneme features from the original phoneme data and obtain a phoneme feature vector.

[0112] In step S303 of some embodiments, the phoneme feature vector is mapped to a preset temporal space through the activation function of the activation layer to obtain the target prediction time feature. The activation function can be a sigmoid activation function, a ReLU activation function, or a tanh activation function, etc., without limitation.

[0113] In step S304 of some embodiments, the initial prediction time features are mapped to a preset mean-variance distribution through a normalization layer to achieve the standardization of the initial prediction time features. The preset mean-variance distribution has a mean of 0 and a variance of 1. In this way, the initial prediction time features can be zero-meaned to obtain the target prediction time features.

[0114] In step S305 of some embodiments, the target prediction time features are mapped to a preset data space through a linear layer, thereby realizing the mapping of the target prediction time features from the vector space to the data space and obtaining a first duration parameter, wherein the first duration parameter can characterize the number of frames for each phoneme.

[0115] Through the above steps S301 to S305, the predicted duration of the original phoneme data, i.e. the first duration parameter, can be obtained relatively easily. This allows the total length of the phoneme hidden layer representation vector to be adjusted according to the first duration parameter, making the speech duration represented by the phoneme hidden layer representation vector more accurate and helping to improve the coherence of the generated speech synthesis data.

[0116] Please see Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S402:

[0117] Step S401: Perform feature recognition on the initial spectral data based on the initial speech embedding vector to obtain background spectral features and key spectral features; wherein, the key spectral features include the speech content corresponding to the original phoneme data, while the background spectral features do not include the speech content corresponding to the original phoneme data.

[0118] Step S402: Filter the background spectral features in the initial spectral data to obtain candidate spectral data.

[0119] In step S401 of some embodiments, the initial speech embedding vector is compared with the features of each frame of the initial spectral data. If the current frame has the same or similar feature information as the initial speech embedding vector, then the current frame is used as the key spectral feature of the initial spectral data. The key spectral feature includes the speech content corresponding to the original phoneme data. Similarly, if the current frame does not have the same or similar feature information as the initial speech embedding vector, then the current frame is used as the background spectral feature of the initial spectral data. The background spectral feature does not include the speech content corresponding to the original phoneme data. The aforementioned feature information includes timbre, pitch, or other speech information.

[0120] In step S402 of some embodiments, background spectral features in the initial spectral data are filtered out, and the retained key spectral features are used as candidate spectral data. For example, the initial spectral data is divided into multiple spectral segments, spectral segments containing background spectral features are removed, and the retained spectral segments containing key spectral features are sequentially spliced ​​together in chronological order to obtain candidate spectral data.

[0121] Through the above steps S401 to S402, useful spectral information in the initial spectral data can be easily identified, spectral data representing background sound can be eliminated, the interference of background noise on speech synthesis can be reduced, the spectral quality of candidate spectral data can be improved, and thus the accuracy of speech synthesis can be improved.

[0122] Please see Figure 5 In some embodiments, the second prediction network includes a second encoding layer and a second decoding layer, and step S105 may include, but is not limited to, steps S501 to S504:

[0123] Step S501: Obtain the second duration parameter of the candidate spectrum data;

[0124] Step S502: The candidate spectral data is encoded using the second coding layer and the second duration parameter to obtain the speech hidden layer representation vector.

[0125] Step S503: Based on the speech object corpus, feature extraction is performed on the speech hidden layer representation vector to obtain the target speech embedding vector;

[0126] Step S504: Attention calculation is performed on the speech hidden layer representation vector through the second decoding layer to obtain intermediate spectrum data.

[0127] In step S501 of some embodiments, candidate spectrum data can be input into a preset time prediction model. The time prediction model performs feature extraction and feature mapping on the candidate spectrum data to obtain the temporal features of the candidate spectrum data in the time domain. Then, the time features are standardized to obtain a second duration parameter. The second duration parameter can characterize the frame number of each phoneme in the candidate spectrum data. The prediction duration represented by the second duration parameter affects the pronunciation length and prosodic characteristics of the generated speech synthesis data. The specific process of obtaining the second duration parameter of the candidate spectrum data is basically the same as the specific process of steps S301 to S305 above, and will not be repeated here.

[0128] In step S502 of some embodiments, the candidate spectrum data is encoded by the second coding layer and the second duration parameter to obtain the phoneme sequence features in the candidate spectrum data and obtain the speech hidden layer representation vector.

[0129] In step S503 of some embodiments, the speech object corpus is traversed, and the speech latent layer representation vector is compared with the speech data of the target speech object in the speech object corpus. The similarity between the speech latent layer representation vector and the candidate speech object embedding vector of the speech data of the target speech object is calculated to obtain a speech similarity value. The candidate speech embedding vector with a speech similarity value greater than or equal to a preset similarity threshold is used as the target speech embedding vector of the target speech object. The candidate speech object embedding vector can be obtained by encoding the speech data of the target speech object through a preset feature encoder. The target speech object includes a speaker, singer, or other speaker with a defined identity, etc. The target speech embedding vector is used to represent the speaking style characteristics of the target speech object in the speech object corpus.

[0130] In step S504 of some embodiments, when performing attention calculation on the speech hidden layer representation vector through the second decoding layer, the query matrix, value matrix, and key matrix of the speech hidden layer representation vector are first calculated. Then, the query matrix, value matrix, and key matrix are weighted using the softmax function to obtain the attention matrix corresponding to the speech hidden layer representation vector. Based on the attention matrix, all feature vectors in the speech hidden layer representation vector are weighted to obtain intermediate spectrum data. Since the background noise interference is eliminated in steps S401 to S402, the accuracy of the intermediate spectrum data is higher than that of the initial spectrum data.

[0131] Through the above steps S501 to S504, the original phoneme data can be predicted again, and the speech feature information and the speech characteristics of the target speaker can be extracted from the candidate spectrum data. This process can further optimize the quality of the spectrum data, obtain intermediate spectrum data with higher accuracy, and also obtain the target speech embedding vector used to characterize the speech style characteristics of the target speaker. In this way, the speech style characteristics of the target speaker can be integrated into the intermediate spectrum data in the subsequent speech synthesis process, thereby improving the accuracy of speech synthesis.

[0132] Please see Figure 6 In some embodiments, step S106 includes, but is not limited to, steps S601 to S602:

[0133] Step S601: Extract features from the target speech embedding vector to obtain the timbre features of the target speaker;

[0134] Step S602: Embed the timbre features into the intermediate spectral data to obtain the target spectral data, which contains the timbre characteristics of the target speaker.

[0135] In step S601 of some embodiments, feature extraction can be performed on the target speech embedding vector by named entity recognition or other methods to obtain all entity features in the target speech embedding vector, including timbre features, pitch features, etc.

[0136] In step S602 of some embodiments, feature embedding can be performed by vector concatenation or vector addition of timbre features and intermediate spectrum data to obtain target spectrum data. For example, the timbre features and intermediate spectrum data are first vectorized, and then the vectorized timbre features and intermediate spectrum data are concatenated or vector added to obtain target spectrum data, so that the target spectrum data can include the timbre characteristics of the target speaker.

[0137] The steps S601 to S602 described above can conveniently extract the timbre features of the target speaker from the target speech embedding vector and fuse the timbre features into the intermediate spectral data, so that the final target spectral data contains the timbre characteristics of the target speaker, thereby improving the accuracy of speech synthesis.

[0138] Please see Figure 7 In some embodiments, the decoding network can be built based on a HiFi-GAN vocoder. The decoding network includes an upsampling module and a residual module. Step S107 may include, but is not limited to, steps S701 to S702:

[0139] Step S701: Upsample the target spectrum data using the upsampling module to obtain initial speech features;

[0140] Step S702: The initial speech features are converted into vocode using the residual module to obtain speech synthesis data.

[0141] In step S701 of some embodiments, the target spectrum data can be upsampled by the upsampling module to achieve transposed convolution of the target spectrum data and obtain initial speech features.

[0142] In step S702 of some embodiments, a residual module can be used to reconstruct the initial speech features to obtain the reconstructed speech waveform, which is then used as speech synthesis data.

[0143] Steps S701 to S702 above can conveniently convert the target spectrum data from the spectrum feature map into waveform features, realize the conversion of the target spectrum data from the frequency domain space to the time domain space, and obtain waveform speech synthesis data, thereby improving the accuracy and efficiency of speech synthesis.

[0144] The speech synthesis method of this application embodiment acquires raw phoneme data, which is text data; inputs the raw phoneme data into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network. Based on the first prediction network and a pre-acquired corpus of the speaking target, a first spectral prediction is performed on the raw phoneme data to obtain initial spectral data and an initial speech embedding vector, enabling preliminary prediction of the raw phoneme data and extraction of phoneme feature information. Based on the initial speech embedding vector, the initial spectral data is filtered to obtain candidate spectral data, which eliminates spectral data representing background noise, reduces interference from background noise on speech synthesis, and improves the spectral quality of the candidate spectral data. Based on the second prediction network and the corpus of the speaking target, a second spectral prediction is performed on the candidate spectral data to obtain intermediate spectral data and a target speech embedding vector, enabling further prediction of the raw phoneme data, further optimizing the quality of the spectral data, obtaining more accurate intermediate spectral data, and also obtaining a target speech embedding vector representing the speaking style characteristics of the target speaking target. Furthermore, the intermediate spectral data is reconstructed based on the target speech embedding vector to obtain the target spectral data. This allows the target speech embedding vector to be fused into the intermediate spectral data, ensuring that the final target spectral data contains the speech style characteristics of the target speaker. Finally, the target spectral data is vocode-converted using a decoding network to obtain speech synthesis data. This process facilitates the conversion of the target spectral data from a spectral feature map to waveform features, achieving the conversion from the frequency domain to the time domain, resulting in waveform-based speech synthesis data. This improves the accuracy and efficiency of speech synthesis.

[0145] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0146] The data acquisition module 801 is used to acquire raw phoneme data, which is text data.

[0147] The input module 802 is used to input the raw phoneme data into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network.

[0148] The first spectrum prediction module 803 is used to perform first spectrum prediction on the original phoneme data based on the first prediction network and the pre-acquired speech object corpus to obtain initial spectrum data and initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking object in the speech object corpus.

[0149] The filtering module 804 is used to filter the initial spectrum data based on the initial speech embedding vector to obtain candidate spectrum data;

[0150] The second spectrum prediction module 805 is used to perform second spectrum prediction on candidate spectrum data based on the second prediction network and the speech object corpus to obtain intermediate spectrum data and target speech embedding vector. The target speech embedding vector is used to characterize the speaking style features of the target speech object in the speech object corpus.

[0151] The reconstruction module 806 is used to reconstruct the intermediate spectral data based on the target speech embedding vector to obtain the target spectral data, which is a Mel spectrogram.

[0152] The vocode conversion module 807 is used to perform vocode conversion on the target spectrum data based on the decoding network to obtain speech synthesis data.

[0153] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0154] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0155] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0156] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0157] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0158] The input / output interface 903 is used to implement information input and output;

[0159] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0160] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0161] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0162] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0163] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium provided in this application acquire raw phoneme data, which is text data. The raw phoneme data is then input into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network. Based on the first prediction network and a pre-acquired corpus of the speaking target, a first spectral prediction is performed on the raw phoneme data to obtain initial spectral data and an initial speech embedding vector. This enables preliminary prediction of the raw phoneme data and extraction of phoneme feature information. Based on the initial speech embedding vector, the initial spectral data is filtered to obtain candidate spectral data. This eliminates spectral data representing background noise, reduces interference from background noise on speech synthesis, and improves the spectral quality of the candidate spectral data. Based on the second prediction network and the corpus of the speaking target, a second spectral prediction is performed on the candidate spectral data to obtain intermediate spectral data and a target speech embedding vector. This enables further prediction of the raw phoneme data, further optimizing the quality of the spectral data, obtaining more accurate intermediate spectral data, and also obtaining a target speech embedding vector representing the speaking style characteristics of the target speaking target. Furthermore, the intermediate spectral data is reconstructed based on the target speech embedding vector to obtain the target spectral data. This allows the target speech embedding vector to be fused into the intermediate spectral data, ensuring that the final target spectral data contains the speech style characteristics of the target speaker. Finally, the target spectral data is vocode-converted using a decoding network to obtain speech synthesis data. This process facilitates the conversion of the target spectral data from a spectral feature map to waveform features, achieving the conversion from the frequency domain to the time domain, resulting in waveform-based speech synthesis data. This improves the accuracy and efficiency of speech synthesis.

[0165] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0166] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0169] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0170] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0172] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The speech synthesis method includes: Obtain raw phoneme data, wherein the raw phoneme data is text data; The raw phoneme data is input into a preset speech synthesis model, which includes a first prediction network, a second prediction network, and a decoding network. Based on the first prediction network and the pre-acquired corpus of speaking objects, the original phoneme data is subjected to a first spectrum prediction to obtain initial spectrum data and an initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking objects in the corpus of speaking objects. The initial spectral data is filtered based on the initial speech embedding vector to obtain candidate spectral data; the candidate spectral data is then subjected to a second spectral prediction based on the second prediction network and the speech object corpus to obtain intermediate spectral data and a target speech embedding vector, wherein the target speech embedding vector is used to characterize the speaking style features of the target speech object in the speech object corpus. The intermediate spectral data is reconstructed based on the target speech embedding vector to obtain target spectral data, which is a Mel spectrogram; the target spectral data is then vocode-converted based on the decoding network to obtain speech synthesis data. The first prediction network includes a first encoding layer and a first decoding layer. The step of performing a first spectral prediction on the original phoneme data based on the first prediction network and a pre-acquired corpus of the speaking audience to obtain initial spectral data and an initial speech embedding vector includes: The process involves: obtaining a first duration parameter from the original phoneme data, whereby the predicted duration represented by the first duration parameter influences the pronunciation length and prosodic characteristics of the generated speech synthesis data; encoding the original phoneme data using the first encoding layer and the first duration parameter to obtain a phoneme hidden layer representation vector; traversing the speech object corpus and comparing the phoneme hidden layer representation vector with the speech data of reference speech objects in the corpus, calculating the similarity between the phoneme hidden layer representation vector and the candidate speech embedding vector of each reference speech object, obtaining multiple feature similarity values, and using the candidate speech embedding vector with a feature similarity value greater than or equal to a preset feature threshold as the initial speech embedding vector of the reference speech object; and performing attention calculation on the phoneme hidden layer representation vector through the first decoding layer to obtain the initial spectral data.

2. The speech synthesis method according to claim 1, characterized in that, The first duration parameter for obtaining the original phoneme data includes: The original phoneme data is input into a preset time prediction model, which includes a convolutional layer, an activation layer, a normalization layer, and a linear layer. The convolutional layer is used to extract features from the original phoneme data to obtain phoneme feature vectors; The phoneme feature vector is mapped to a preset temporal space through the activation layer to obtain the initial predicted temporal features; The initial prediction time features are standardized by the normalization layer to obtain the target prediction time features. The target predicted time features are mapped to a preset data space through the linear layer to obtain the first duration parameter.

3. The speech synthesis method according to claim 1, characterized in that, The second prediction network includes a second encoding layer and a second decoding layer. The step of performing a second spectral prediction on the candidate spectral data based on the second prediction network and the speech object corpus to obtain intermediate spectral data and a target speech embedding vector includes: Obtain the second duration parameter of the candidate spectrum data; The candidate spectral data is encoded using the second coding layer and the second duration parameter to obtain a speech hidden layer representation vector; Based on the corpus of the speaking object, feature extraction is performed on the speech hidden layer representation vector to obtain the target speech embedding vector; The intermediate spectral data is obtained by performing attention calculation on the speech hidden layer representation vector through the second decoding layer.

4. The speech synthesis method according to claim 1, characterized in that, The step of filtering the initial spectral data based on the initial speech embedding vector to obtain candidate spectral data includes: Based on the initial speech embedding vector, feature recognition is performed on the initial spectral data to obtain background spectral features and key spectral features; wherein, the key spectral features include the speech content corresponding to the original phoneme data, and the background spectral features do not include the speech content corresponding to the original phoneme data; The background spectral features in the initial spectral data are filtered to obtain the candidate spectral data.

5. The speech synthesis method according to claim 1, characterized in that, The process of reconstructing the intermediate spectral data based on the target speech embedding vector to obtain the target spectral data includes: Feature extraction is performed on the target speech embedding vector to obtain the timbre features of the target speaker; The timbre features are embedded into the intermediate spectral data to obtain the target spectral data, which contains the timbre characteristics of the target speaker.

6. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The decoding network includes an upsampling module and a residual module. The step of performing vocode conversion on the target spectral data based on the decoding network to obtain speech synthesis data includes: The target spectral data is upsampled using the upsampling module to obtain initial speech features. The speech synthesis data is obtained by performing vocode conversion on the initial speech features using the residual module.

7. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire raw phoneme data, which is text data; An input module is used to input the raw phoneme data into a preset speech synthesis model, the speech synthesis model including a first prediction network, a second prediction network, and a decoding network; The first spectrum prediction module is used to perform a first spectrum prediction on the original phoneme data based on the first prediction network and the pre-acquired speech object corpus to obtain initial spectrum data and an initial speech embedding vector. The initial speech embedding vector is used to characterize the speaking style characteristics of the reference speaking object in the speech object corpus. The filtering module is used to filter the initial spectrum data based on the initial speech embedding vector to obtain candidate spectrum data; The second spectrum prediction module is used to perform a second spectrum prediction on the candidate spectrum data based on the second prediction network and the speech object corpus to obtain intermediate spectrum data and target speech embedding vector. The target speech embedding vector is used to characterize the speaking style features of the target speech object in the speech object corpus. The reconstruction module is used to reconstruct the intermediate spectral data based on the target speech embedding vector to obtain target spectral data, which is a Mel spectrogram. A vocode conversion module is used to perform vocode conversion on the target spectrum data based on the decoding network to obtain speech synthesis data; The first prediction network includes a first encoding layer and a first decoding layer. The step of performing a first spectral prediction on the original phoneme data based on the first prediction network and a pre-acquired corpus of the speaking audience to obtain initial spectral data and an initial speech embedding vector includes: The process involves: obtaining a first duration parameter from the original phoneme data, whereby the predicted duration represented by the first duration parameter influences the pronunciation length and prosodic characteristics of the generated speech synthesis data; encoding the original phoneme data using the first encoding layer and the first duration parameter to obtain a phoneme hidden layer representation vector; traversing the speech object corpus and comparing the phoneme hidden layer representation vector with the speech data of reference speech objects in the corpus, calculating the similarity between the phoneme hidden layer representation vector and the candidate speech embedding vector of each reference speech object, obtaining multiple feature similarity values, and using the candidate speech embedding vector with a feature similarity value greater than or equal to a preset feature threshold as the initial speech embedding vector of the reference speech object; and performing attention calculation on the phoneme hidden layer representation vector through the first decoding layer to obtain the initial spectral data.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images