Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining original text data and reference speech data of a reference speaker; inputting the original text data and the reference speech data into a speech synthesis model; encoding the original text data to obtain an original text encoding vector; extracting features from the reference speech data to obtain a reference timbre feature vector; performing vocabulary recognition processing on the reference speech data to obtain a reference word embedding vector; aligning the original text encoding vector, the reference timbre feature vector and the reference word embedding vector to obtain a target phoneme encoding vector; performing time prediction processing on the target phoneme encoding vector to obtain a duration parameter; and performing speech synthesis processing on the target phoneme encoding vector according to the duration parameter to obtain speech synthesis data. The application can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, speech synthesis device, electronic device and storage medium. Background Technology

[0002] Currently, the accuracy of most speech synthesis methods is often limited by the length of the text data to be processed. When the text data is long, it is often impossible to synthesize coherent speech data, resulting in poor speech synthesis accuracy. Therefore, how to improve the accuracy of speech synthesis has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a speech synthesis method, speech synthesis device, electronic device, and storage medium, which aim to improve the accuracy of speech synthesis.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] Obtain the preset raw text data and the reference speech data of the reference speaker;

[0006] The original text data and the reference speech data are input into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network.

[0007] The original text data is encoded using the phoneme coding network to obtain the original text coding vector.

[0008] The speech embedding network is used to extract features from the reference speech data to obtain a reference timbre feature vector, which is used to characterize the prosodic features of the reference speaker.

[0009] The reference speech data is processed by the language representation network to obtain a reference word embedding vector, which is used to represent the language features of the reference speech data.

[0010] The original text encoding vector, the reference timbre feature vector, and the reference word embedding vector are aligned to obtain the target phoneme encoding vector.

[0011] The target phoneme encoding vector is subjected to time prediction processing by the time prediction network to obtain the duration parameter;

[0012] The target phoneme encoding vector is processed by the decoding network and the duration parameter to obtain speech synthesis data.

[0013] In some embodiments, encoding the original text data through the phoneme coding network to obtain the original text coding vector includes:

[0014] The original text data is converted using the phoneme encoding network and a preset reference dictionary to obtain the original phoneme data.

[0015] The original phoneme data is encoded using the phoneme coding network to obtain the original text coding vector.

[0016] In some embodiments, the speech embedding network includes a feature extraction layer and an LSTM layer. The step of extracting features from the reference speech data using the speech embedding network to obtain a reference timbre feature vector includes:

[0017] The prosodic features are extracted from the reference speech data through the feature extraction layer to obtain speech prosodic features;

[0018] The speech prosody features are encoded using the LSTM layer to obtain the reference timbre feature vector.

[0019] In some embodiments, the language representation network includes a BERT layer and a first activation function, and the step of performing lexical recognition processing on the reference speech data through the language representation network to obtain reference word embedding vectors includes:

[0020] The reference speech data is lexical encoded using the BERT layer to obtain reference speech word vectors;

[0021] The reference speech word vector is obtained by activating the reference speech word vector using the first activation function.

[0022] In some embodiments, aligning the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector includes:

[0023] The original text encoding vector, the reference timbre feature vector, and the reference word embedding vector are concatenated to obtain the initial phoneme encoding vector.

[0024] The initial phoneme encoding vector is downsampled according to preset sampling parameters to obtain candidate phoneme encoding vectors;

[0025] The candidate phoneme encoding vector is activated by a preset second activation function to obtain the target phoneme encoding vector.

[0026] In some embodiments, the time prediction network includes convolutional layers, activation layers, normalization layers, dropout layers, and linear layers. The step of performing time prediction processing on the target phoneme encoding vector through the time prediction network to obtain a duration parameter includes:

[0027] The target phoneme encoding vector is convolved by the convolutional layer to obtain the initial phoneme convolutional vector.

[0028] The initial phoneme convolution vector is activated by the activation layer to obtain the target phoneme convolution vector;

[0029] The target phoneme convolution vector is normalized using the normalization process to obtain candidate temporal features;

[0030] The candidate time features are filtered through the discard layer to obtain the target time features;

[0031] The target time feature is mapped to a preset data space through the linear layer to obtain the duration parameter.

[0032] In some embodiments, the step of performing speech synthesis processing on the target phoneme encoding vector through the decoding network and the duration parameter to obtain synthesized speech data includes:

[0033] The target phoneme encoding vector is decoded by the decoding network to obtain a speech synthesis sequence;

[0034] The speech synthesis sequence is lengthened by adjusting the duration parameter to obtain the speech synthesis data.

[0035] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0036] The data acquisition module is used to acquire preset raw text data and reference speech data of the reference speaker;

[0037] The input module is used to input the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network.

[0038] The text encoding module is used to encode the original text data through the phoneme encoding network to obtain the original text encoding vector;

[0039] The feature extraction module is used to extract features from the reference speech data through the speech embedding network to obtain a reference timbre feature vector, which is used to characterize the prosodic features of the reference speaker.

[0040] The language recognition module is used to perform vocabulary recognition processing on the reference speech data through the language representation network to obtain a reference word embedding vector, which is used to represent the language characteristics of the reference speech data.

[0041] The alignment module is used to align the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector.

[0042] The time prediction module is used to perform time prediction processing on the target phoneme encoding vector through the time prediction network to obtain the duration parameter;

[0043] The speech synthesis module is used to perform speech synthesis processing on the target phoneme encoding vector through the decoding network and the duration parameter to obtain synthesized speech data.

[0044] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application acquire preset raw text data and reference speech data of a reference speaker; input the raw text data and reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. The phoneme coding network encodes the raw text data to obtain a raw text coding vector, which effectively improves the standardization of the text. The speech embedding network extracts features from the reference speech data to obtain a reference timbre feature vector, which is used to represent the prosodic characteristics of the reference speaker. This method enhances the extraction of important feature information from the reference speech data, obtains sentence-level language features, and improves the accuracy of the reference timbre feature vector. Furthermore, the language representation network performs lexical recognition processing on the reference speech data to obtain a reference word embedding vector, which is used to represent the language characteristics of the reference speech data, effectively identifying word-level language features in the reference speech data and improving the diversity of language features. Furthermore, the original text encoding vector, reference timbre feature vector, and reference word embedding vector are aligned to obtain the target phoneme encoding vector. This allows for the convenient integration of the prosodic and linguistic features of the reference object into the original text encoding vector, ensuring that the target phoneme encoding vector conforms to the current language style. Further, a time prediction network is used to perform time prediction processing on the target phoneme encoding vector to obtain a duration parameter. A decoding network and the duration parameter are then used to perform speech synthesis processing on the target phoneme encoding vector to obtain speech synthesis data. This ensures that the speech synthesis data corresponding to the original text data contains the linguistic features of the reference object, improving the accuracy of speech synthesis. Attached Figure Description

[0047] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 4 yes Figure 1 The flowchart of step S105 in the process;

[0051] Figure 5 yes Figure 1 The flowchart of step S106 in the process;

[0052] Figure 6 yes Figure 1 The flowchart of step S107 in the process;

[0053] Figure 7 yes Figure 1 The flowchart of step S108 in the process;

[0054] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0063] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0064] Phoneme: The smallest unit of speech based on the natural properties of speech. It is analyzed based on the articulation of a syllable, and one articulation constitutes one phoneme.

[0065] Activation function: This is a function that runs on the neurons of an artificial neural network and is responsible for mapping the input of the neuron to its output.

[0066] Encoder: Transforms an input sequence into a fixed-length vector.

[0067] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0068] BERT (Bidirectional Encoder Representation from Transformers) model: The BERT model further enhances the generalization ability of word embedding models, fully describing character-level, word-level, sentence-level, and even inter-sentence relationship features, and is built based on Transformers. BERT has three types of embeddings: Token Embedding, Segment Embedding, and Position Embedding. Token Embeddings are word vectors, with the first word as a CLS marker, which can be used for subsequent classification tasks. Segment Embeddings are used to distinguish between two types of sentences because pre-training involves not only LM but also classification tasks with two sentences as input. Position Embeddings, where the positional word vectors are not trigonometric functions from Transformer, but rather learned by BERT during training. However, BERT directly trains a position embedding to preserve positional information. A vector is randomly initialized for each position and added to the model training, resulting in an embedding containing positional information. Finally, BERT chooses to directly concatenate this position embedding with the word embedding.

[0069] Speech synthesis refers to the synthesis of intelligible and natural speech from text, also known as text-to-speech (TTS).

[0070] Currently, the accuracy of most speech synthesis methods is often limited by the length of the text data to be processed. When the text data is long, it is often impossible to synthesize coherent speech data, resulting in poor speech synthesis accuracy. Therefore, how to improve the accuracy of speech synthesis has become an urgent technical problem to be solved.

[0071] Based on this, embodiments of this application provide a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium, aiming to improve the accuracy of speech synthesis.

[0072] The speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0073] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0074] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0075] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0076] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0077] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0078] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0079] Step S101: Obtain the preset raw text data and reference speech data of the reference speaker;

[0080] Step S102: Input the original text data and reference speech data into the preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network.

[0081] Step S103: Encode the original text data using a phoneme coding network to obtain the original text coding vector;

[0082] Step S104: Extract features from the reference speech data through a speech embedding network to obtain a reference timbre feature vector. The reference timbre feature vector is used to characterize the prosodic features of the reference speaker.

[0083] Step S105: The reference speech data is processed by a language representation network to obtain a reference word embedding vector. The reference word embedding vector is used to represent the language features of the reference speech data.

[0084] Step S106: Align the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector;

[0085] Step S107: Perform time prediction processing on the target phoneme encoding vector through a time prediction network to obtain the duration parameter;

[0086] Step S108: The target phoneme encoding vector is processed by speech synthesis through a decoding network and duration parameters to obtain speech synthesis data.

[0087] Steps S101 to S108 of this embodiment involve acquiring preset original text data and reference speech data of a reference speaker; inputting the original text data and reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. Encoding the original text data through the phoneme coding network yields an original text coding vector, effectively improving the text's standardization. Feature extraction from the reference speech data through the speech embedding network yields a reference timbre feature vector, which represents the prosodic characteristics of the reference speaker. This method enhances the extraction of important feature information from the reference speech data, obtains sentence-level language features, and improves the accuracy of the reference timbre feature vector. Furthermore, lexical recognition processing is performed on the reference speech data through the language representation network to obtain a reference word embedding vector, which represents the language characteristics of the reference speech data. This effectively identifies word-level language features in the reference speech data, improving the diversity of language features. Furthermore, the original text encoding vector, reference timbre feature vector, and reference word embedding vector are aligned to obtain the target phoneme encoding vector. This allows for the convenient integration of the prosodic and linguistic features of the reference object into the original text encoding vector, ensuring that the target phoneme encoding vector conforms to the current language style. Further, a time prediction network is used to perform time prediction processing on the target phoneme encoding vector to obtain a duration parameter. A decoding network and the duration parameter are then used to perform speech synthesis processing on the target phoneme encoding vector to obtain speech synthesis data. This ensures that the speech synthesis data corresponding to the original text data contains the linguistic features of the reference object, improving the accuracy of speech synthesis.

[0088] In step S101 of some embodiments, the raw text data can be obtained from a public dataset, or from an existing text database or network platform, without limitation. For example, the public dataset could be the THCHS30 dataset or the LJSpeech dataset, etc. The raw text data includes raw text with a length greater than a preset text length. This preset text length can be set according to actual conditions; for example, a preset text length of 10,000 bytes means that text with a length greater than 10,000 bytes is considered long text, and text with a length less than or equal to 10,000 bytes is considered short text. This application embodiment can perform speech synthesis on raw text with a length greater than the preset text length (i.e., long text), improving the accuracy and fluency of speech synthesis.

[0089] Simultaneously, by writing web crawlers and setting up data sources, targeted data can be scraped to obtain reference speech data of the speaker. The data source can be various types of online platforms, social media, or specific audio databases. The reference speaker can be an internet user, speaker, singer, etc., and the reference speech data can be the speaker's music, presentation, chat dialogue, etc. This method allows for relatively convenient acquisition of reference speech data and raw text data, improving data acquisition efficiency.

[0090] In step S102 of some embodiments, the original text data and reference speech data can be input into a preset speech synthesis model through a preset script program or other computer program. The speech synthesis model can be based on an encoder-decoder structure and includes a phoneme encoding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. The phoneme encoding network is mainly used to encode the input original text data, extract phoneme features from the original text data, and obtain the original text encoding vector. The speech embedding network is mainly used to encode the reference speech data, extract the speech characteristics of the reference speaker, and obtain a reference timbre feature vector representing the prosodic characteristics of the reference speaker. The language representation network is mainly used to extract word-level features from the reference speech data, and obtain a reference word embedding vector representing the language characteristics of the reference speech data. The time prediction network is mainly used to predict the frame number of phoneme features in the original text data, and obtain a duration parameter. The decoding network is mainly used to decode the target phoneme encoding vector, obtain the corresponding target Mel spectrum, and perform vocode conversion on the target Mel spectrum to obtain the speech synthesis data corresponding to the original text data. This speech synthesis model can effectively embed the speech characteristics of the reference speaker into the speech synthesis data, enabling the speech synthesis data to contain the required emotional and prosodic features, thereby improving the accuracy of speech synthesis.

[0091] It should be noted that when training a speech synthesis model, multiple short text data (i.e., texts with a length less than or equal to a preset text length) can be concatenated to connect coherent sentences in the short text dataset, forming fixed-length text data. This fixed-length text data is then used as long text data, and the speech synthesis model is trained based on this long text data.

[0092] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S202:

[0093] Step S201: The original text data is converted using a phoneme encoding network and a preset reference dictionary to obtain the original phoneme data;

[0094] Step S202: The original phoneme data is encoded using a phoneme coding network to obtain the original text coding vector.

[0095] In step S201 of some embodiments, the preset reference dictionary may be a CMUSphinx data dictionary, which contains multiple words. A word list can be constructed using the phoneme coding network and the reference dictionary. The word list is formatted as one word or character per line. This word list can be used to perform data conversion on the text content in the original text data, converting the words or characters corresponding to the text content into phoneme sequences, thereby forming the original phoneme data corresponding to the original text data.

[0096] In step S202 of some embodiments, the original phoneme data is encoded by a phoneme coding network to convert the phoneme sequence in text form into a vector form, thereby obtaining the original text coding vector.

[0097] By performing data conversion on the original text data through the above steps S201 to S202, the standardization of the text can be effectively improved, and the phoneme features in the original text data can be extracted more easily, processing the original text data into a vector form of the original text encoding vector.

[0098] Please see Figure 3 In some embodiments, the speech embedding network includes a feature extraction layer and an LSTM layer, and step S104 may include, but is not limited to, steps S301 to S302:

[0099] Step S301: Prosodic features are extracted from the reference speech data through the feature extraction layer to obtain speech prosodic features;

[0100] Step S302: Encode the speech prosody features through an LSTM layer to obtain a reference timbre feature vector.

[0101] In step S301 of some embodiments, the speech embedding network can be constructed based on a feature encoder, which includes multiple feedforward transformers for feature extraction. The speech embedding network includes a feature extraction layer and an LSTM layer. The feature extraction layer performs convolution processing on the speech features of the reference speech data to extract the frequency domain spatial features of the reference speech data. The extracted frequency domain spatial features are then filtered according to the feature type to obtain speech prosodic features, which include timbre, pitch, tone, and other feature information of the reference data object.

[0102] In step S302 of some embodiments, the speech prosodic features can be encoded from left to right using an LSTM layer to obtain a first encoding vector, and then encoded from right to left using another LSTM layer to obtain a second encoding vector. Further, a reference timbre feature vector is obtained by concatenating the first encoding vector, the second encoding vector, and the state vector of the LSTM layer. This reference timbre feature vector can be used to characterize the prosodic features of a reference speaker.

[0103] Through the above steps S301 to S302, feature extraction and encoding processing of reference speech data can be performed based on the feature encoder. This makes it easier to obtain prosodic feature information in the reference speech data, enhances the extraction of important feature information in the reference speech data, obtains linguistic features at the sentence level, and improves the accuracy of the reference timbre feature vector.

[0104] Please see Figure 4 In some embodiments, the language representation network includes a BERT layer and a first activation function, and step S105 may include, but is not limited to, steps S401 to S402:

[0105] Step S401: The reference speech data is lexical encoded using the BERT layer to obtain reference speech word vectors;

[0106] Step S402: The reference speech word vector is activated by the first activation function to obtain the reference word embedding vector.

[0107] In step S401 of some embodiments, the reference speech data can be segmented using a BERT layer and basic grammar rules to obtain multiple speech segments. Then, an existing phoneme lookup table is consulted, and based on the correspondence between phonemes and characters in the phoneme lookup table, the corresponding word segments are obtained. Furthermore, the BERT layer is used to perform lexical encoding on the word segments corresponding to this series of speech segments to obtain reference speech word vectors.

[0108] In step S402 of some embodiments, the preset first activation function includes ReLU activation function, Sigmoid activation function, etc. Taking ReLU activation function as an example, when activating the reference speech word vector using ReLU activation function, the reference speech word vector is first moved to the activation region, and then mapped in the activation region to map the reference speech word vector to a fixed-dimensional vector space, resulting in a reference word embedding vector. The vector dimension of this fixed-dimensional vector space is lower than the vector dimension of the reference speech word vector. The reference word embedding vector is used to represent the linguistic characteristics of the reference speech data.

[0109] Through the above steps S401 to S402, word-level language features in the reference speech data can be effectively identified, thereby improving the diversity of language features.

[0110] Please see Figure 5 In some embodiments, step S106 may include, but is not limited to, steps S501 to S503:

[0111] Step S501: The original text encoding vector, the reference timbre feature vector, and the reference word embedding vector are concatenated to obtain the initial phoneme encoding vector.

[0112] Step S502: The initial phoneme encoding vector is downsampled according to the preset sampling parameters to obtain the candidate phoneme encoding vector;

[0113] Step S503: The candidate phoneme encoding vector is activated by a preset second activation function to obtain the target phoneme encoding vector.

[0114] In step S501 of some embodiments, when concatenating the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector, the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector can be vectorized to obtain an initial phoneme encoding vector. The vector length of the initial phoneme encoding vector can be three times the vector length of the original text encoding vector.

[0115] In step S502 of some embodiments, the preset sampling parameters can be set according to actual conditions and are not limited. The initial phoneme encoding vector is downsampled using methods such as max pooling or average pooling according to the preset sampling parameters to retain important element features in the initial phoneme encoding vector, reduce the data volume, and obtain candidate phoneme encoding vectors.

[0116] In step S503 of some embodiments, the preset second activation function includes ReLU activation function, Sigmoid activation function, etc. Taking ReLU activation function as an example, when activating the candidate phoneme encoding vector using ReLU activation function, the candidate phoneme encoding vector is first moved to the activation region, and then mapped in the activation region to map the candidate phoneme encoding vector to a fixed-dimensional vector space to obtain the target phoneme encoding vector. The vector dimension of this fixed-dimensional vector space is lower than the vector dimension of the candidate phoneme encoding vector.

[0117] Steps S501 to S503 can conveniently achieve attention alignment of the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector, enabling the reference timbre feature vector and the reference word embedding vector to be embedded into the original text encoding vector to obtain the target phoneme encoding vector. This allows the target phoneme encoding vector to contain the prosodic and linguistic features of the reference object, thereby making the target phoneme encoding vector conform to the needs of the current language style.

[0118] Please see Figure 6 In some embodiments, the time prediction network includes convolutional layers, activation layers, normalization layers, dropout layers, and linear layers. Step S107 includes, but is not limited to, steps S601 to S605:

[0119] Step S601: The target phoneme encoding vector is convolved by a convolutional layer to obtain the initial phoneme convolutional vector;

[0120] Step S602: The initial phoneme convolution vector is activated by the activation layer to obtain the target phoneme convolution vector;

[0121] Step S603: Normalize the target phoneme convolution vector to obtain candidate temporal features;

[0122] Step S604: The candidate time features are filtered through a discard layer to obtain the target time features;

[0123] Step S605: Map the target time features to a preset data space through a linear layer to obtain the duration parameter.

[0124] In step S601 of some embodiments, the target phoneme encoding vector can be convolved by a convolutional layer to obtain important phoneme feature information in the target phoneme encoding vector and obtain an initial phoneme convolutional vector.

[0125] In step S602 of some embodiments, when activating the initial phoneme convolution vector using the activation function of the activation layer, the activation function includes the ReLU activation function, the Sigmoid activation function, etc. Taking the ReLU activation function as an example, when activating the initial phoneme convolution vector using the ReLU activation function, the initial phoneme convolution vector is first moved to the activation region, and then mapped in the activation region, mapping the initial phoneme convolution vector from the high-dimensional vector space to the low-dimensional vector space to obtain the target phoneme convolution vector.

[0126] In step S603 of some embodiments, when normalizing the target phoneme convolution vector, the mean and variance of a series of target phoneme convolution vectors can be calculated to obtain the vector mean and vector variance values. Based on the vector mean and vector variance values, the target phoneme convolution vector is normalized to a preset distribution, where the vector mean is 0 and the vector variance is 1. That is, normalizing the target phoneme convolution vector achieves zero-mean processing of the target phoneme convolution vector, thus obtaining candidate temporal features.

[0127] In step S604 of some embodiments, the reference time features of the discard layer are compared with the candidate time features. Based on the comparison results, the candidate time features are filtered, and the candidate time features whose feature difference with the reference time features is less than a preset threshold are selected. The candidate time features that meet the requirements are used as the target time features.

[0128] In step S605 of some embodiments, the target temporal features are mapped to a preset data space through a linear layer, realizing the mapping of the target temporal features from the vector space to the data space, and obtaining a duration parameter, wherein the duration parameter can characterize the number of frames for each phoneme. The prediction duration characterized by the duration parameter directly affects the pronunciation length and prosodic characteristics of the final generated speech synthesis data.

[0129] Through the above steps S601 to S605, the predicted duration of the original text data, i.e. the duration parameter, can be obtained relatively easily. This allows the total length of the speech synthesis sequence to be adjusted according to the duration parameter, making the speech duration of the speech synthesis data more accurate, thereby improving the prosodic authenticity and coherence of the speech synthesis data.

[0130] Please see Figure 7 In some embodiments, step S108 may include, but is not limited to, steps S701 to S702:

[0131] Step S701: The target phoneme encoding vector is decoded by a decoding network to obtain a speech synthesis sequence;

[0132] Step S702: Adjust the length of the speech synthesis sequence using the duration parameter to obtain speech synthesis data.

[0133] In step S701 of some embodiments, when decoding the target phoneme encoding vector through the decoding network, the latent vector features in the target phoneme encoding vector are first extracted through the decoding network. Based on the latent vector features, the target phoneme encoding vector is converted from vector form to sequence form to obtain a speech synthesis sequence. The speech synthesis sequence can be in the form of a Mel-cepstrum. The speech information of the speech synthesis sequence corresponds to the text information of the original text data.

[0134] In step S702 of some embodiments, the length of the speech synthesis sequence is adjusted according to the duration parameter to control the total length of the speech synthesis sequence to be consistent with the number of phoneme frames in the duration parameter, thereby obtaining speech synthesis data.

[0135] The steps S701 to S702 described above perform speech synthesis on the target phoneme encoding vector based on the decoding network and duration parameters. This can effectively convert the target phoneme encoding vector into a speech synthesis sequence. Furthermore, by adjusting the total length of the speech synthesis sequence according to the predicted duration of the duration parameter, the speech duration of the speech synthesis data becomes more accurate, thereby improving the prosodic authenticity and coherence of the speech synthesis data and ultimately enhancing the accuracy of speech synthesis.

[0136] The speech synthesis method of this application acquires preset original text data and reference speech data of a reference speaker; the original text data and reference speech data are input into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. The phoneme coding network encodes the original text data to obtain an original text coding vector, which effectively improves the standardization of the text. The speech embedding network extracts features from the reference speech data to obtain a reference timbre feature vector, which is used to represent the prosodic characteristics of the reference speaker. This method enhances the extraction of important feature information in the reference speech data, obtains sentence-level language features, and improves the accuracy of the reference timbre feature vector. Furthermore, the language representation network performs lexical recognition processing on the reference speech data to obtain a reference word embedding vector, which is used to represent the language characteristics of the reference speech data. This effectively identifies word-level language features in the reference speech data and improves the diversity of language features. Furthermore, the original text encoding vector, reference timbre feature vector, and reference word embedding vector are aligned to obtain the target phoneme encoding vector. This allows for the convenient integration of the prosodic and linguistic features of the reference object into the original text encoding vector, ensuring that the target phoneme encoding vector conforms to the current language style. Further, a time prediction network is used to perform time prediction processing on the target phoneme encoding vector to obtain a duration parameter. A decoding network and the duration parameter are then used to perform speech synthesis processing on the target phoneme encoding vector to obtain speech synthesis data. This ensures that the speech synthesis data corresponding to the original text data contains the linguistic features of the reference object, improving the accuracy of speech synthesis.

[0137] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0138] The data acquisition module 801 is used to acquire preset raw text data and reference speech data of the reference speaker;

[0139] The input module 802 is used to input the original text data and reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network.

[0140] The text encoding module 803 is used to encode the original text data through a phoneme encoding network to obtain the original text encoding vector;

[0141] The feature extraction module 804 is used to extract features from the reference speech data through the speech embedding network to obtain a reference timbre feature vector, which is used to characterize the prosodic features of the reference speaker.

[0142] The language recognition module 805 is used to perform vocabulary recognition processing on the reference speech data through a language representation network to obtain reference word embedding vectors, which are used to represent the language features of the reference speech data.

[0143] Alignment module 806 is used to align the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector;

[0144] The time prediction module 807 is used to perform time prediction processing on the target phoneme encoding vector through the time prediction network to obtain the duration parameter;

[0145] The speech synthesis module 808 is used to perform speech synthesis processing on the target phoneme encoding vector through a decoding network and duration parameters to obtain synthesized speech data.

[0146] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0147] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0148] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0149] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0150] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0151] The input / output interface 903 is used to implement information input and output;

[0152] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0153] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0154] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0155] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0156] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0157] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium provided in this application acquire preset original text data and reference speech data of a reference speaker; input the original text data and reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. Encoding the original text data through the phoneme coding network yields an original text coding vector, which effectively improves the standardization of the text. Feature extraction from the reference speech data through the speech embedding network yields a reference timbre feature vector, which is used to represent the prosodic characteristics of the reference speaker. This method enhances the extraction of important feature information from the reference speech data, obtains sentence-level language features, and improves the accuracy of the reference timbre feature vector. Furthermore, lexical recognition processing is performed on the reference speech data through the language representation network to obtain a reference word embedding vector, which is used to represent the language characteristics of the reference speech data. This effectively identifies word-level language features in the reference speech data and improves the diversity of language features. Furthermore, the original text encoding vector, reference timbre feature vector, and reference word embedding vector are aligned to obtain the target phoneme encoding vector. This allows for the convenient integration of the prosodic and linguistic features of the reference object into the original text encoding vector, ensuring that the target phoneme encoding vector conforms to the current language style. Further, a time prediction network is used to perform time prediction processing on the target phoneme encoding vector to obtain a duration parameter. A decoding network and the duration parameter are then used to perform speech synthesis processing on the target phoneme encoding vector to obtain speech synthesis data. This ensures that the speech synthesis data corresponding to the original text data contains the linguistic features of the reference object, improving the accuracy of speech synthesis.

[0158] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0159] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0162] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0163] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the preset raw text data and the reference speech data of the reference speaker; The original text data and the reference speech data are input into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. The original text data is encoded using the phoneme coding network to obtain the original text coding vector. The speech embedding network is used to extract features from the reference speech data to obtain a reference timbre feature vector, which is used to characterize the prosodic features of the reference speaker. The reference speech data is processed by the language representation network to obtain a reference word embedding vector, which is used to represent the language features of the reference speech data. The original text encoding vector, the reference timbre feature vector, and the reference word embedding vector are aligned to obtain the target phoneme encoding vector. The target phoneme encoding vector is subjected to time prediction processing by the time prediction network to obtain the duration parameter; The target phoneme encoding vector is processed by the decoding network and the duration parameter to obtain speech synthesis data; The time prediction network includes convolutional layers, activation layers, normalization layers, dropout layers, and linear layers. The time prediction process performed on the target phoneme encoding vector using the time prediction network to obtain a duration parameter includes: The target phoneme encoding vector is convolved by the convolutional layer to obtain an initial phoneme convolutional vector; the initial phoneme convolutional vector is activated by the activation layer to obtain a target phoneme convolutional vector; the target phoneme convolutional vector is normalized by the normalization layer to obtain candidate time features; the reference time features of the dropout layer are compared with the candidate time features, and the candidate time features are filtered according to the comparison results, and the candidate time features with a feature difference of less than a preset threshold are selected as target time features; the target time features are mapped to a preset data space by the linear layer to obtain the duration parameter.

2. The method according to claim 1, characterized in that, The process of encoding the original text data through the phoneme coding network to obtain the original text encoding vector includes: The original text data is converted using the phoneme encoding network and a preset reference dictionary to obtain the original phoneme data. The original phoneme data is encoded using the phoneme coding network to obtain the original text coding vector.

3. The method according to claim 1, characterized in that, The speech embedding network includes a feature extraction layer and an LSTM layer. The step of extracting features from the reference speech data using the speech embedding network to obtain a reference timbre feature vector includes: The prosodic features are extracted from the reference speech data through the feature extraction layer to obtain speech prosodic features; The speech prosody features are encoded using the LSTM layer to obtain the reference timbre feature vector.

4. The method according to claim 1, characterized in that, The language representation network includes a BERT layer and a first activation function. The step of performing vocabulary recognition processing on the reference speech data through the language representation network to obtain reference word embedding vectors includes: The reference speech data is lexical encoded using the BERT layer to obtain reference speech word vectors; The reference speech word vector is obtained by activating the reference speech word vector using the first activation function.

5. The method according to claim 1, characterized in that, The alignment process of the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector includes: The original text encoding vector, the reference timbre feature vector, and the reference word embedding vector are concatenated to obtain the initial phoneme encoding vector. The initial phoneme encoding vector is downsampled according to preset sampling parameters to obtain candidate phoneme encoding vectors; The candidate phoneme encoding vector is activated by a preset second activation function to obtain the target phoneme encoding vector.

6. The method according to any one of claims 1 to 5, characterized in that, The step of performing speech synthesis processing on the target phoneme encoding vector through the decoding network and the duration parameter to obtain synthesized speech data includes: The target phoneme encoding vector is decoded by the decoding network to obtain a speech synthesis sequence; The speech synthesis sequence is lengthened by adjusting the duration parameter to obtain the speech synthesis data.

7. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire preset raw text data and reference speech data of the reference speaker; The input module is used to input the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a phoneme coding network, a speech embedding network, a language representation network, a time prediction network, and a decoding network. The text encoding module is used to encode the original text data through the phoneme encoding network to obtain the original text encoding vector; The feature extraction module is used to extract features from the reference speech data through the speech embedding network to obtain a reference timbre feature vector, which is used to characterize the prosodic features of the reference speaker. The language recognition module is used to perform vocabulary recognition processing on the reference speech data through the language representation network to obtain a reference word embedding vector, which is used to represent the language characteristics of the reference speech data. The alignment module is used to align the original text encoding vector, the reference timbre feature vector, and the reference word embedding vector to obtain the target phoneme encoding vector. The time prediction module is used to perform time prediction processing on the target phoneme encoding vector through the time prediction network to obtain the duration parameter; The speech synthesis module is used to perform speech synthesis processing on the target phoneme encoding vector through the decoding network and the duration parameter to obtain synthesized speech data; The time prediction network includes convolutional layers, activation layers, normalization layers, dropout layers, and linear layers. The time prediction process performed on the target phoneme encoding vector using the time prediction network to obtain a duration parameter includes: The target phoneme encoding vector is convolved by the convolutional layer to obtain an initial phoneme convolutional vector; the initial phoneme convolutional vector is activated by the activation layer to obtain a target phoneme convolutional vector; the target phoneme convolutional vector is normalized by the normalization layer to obtain candidate time features; the reference time features of the dropout layer are compared with the candidate time features, and the candidate time features are filtered according to the comparison results, and the candidate time features with a feature difference of less than a preset threshold are selected as target time features; the target time features are mapped to a preset data space by the linear layer to obtain the duration parameter.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images