Speech synthesis method, speech synthesis device, electronic device and storage medium

Abstract

The present application provides a speech synthesis method, speech synthesis device, electronic device and storage medium, belonging to the field of artificial intelligence technology. The method includes: obtaining original text data to be processed and reference speech data of a reference speaker; inputting the original text data and reference speech data into a speech synthesis model, the speech synthesis model including a text encoding network, a speech encoding network, a speech alignment network and a decoding network; encoding the original text data through the text encoding network to obtain an original text embedding vector; encoding the reference speech data through the speech encoding network to obtain a reference speech embedding vector; aligning the original text embedding vector, the reference speech embedding vector and the text style embedding vector through the speech alignment network to obtain a target text embedding vector; decoupling the target text embedding vector through the decoding network to obtain synthesized speech data. The present application improves the accuracy of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS), involves synthesizing understandable, natural speech from text. Common speech synthesis methods typically train models based on fixed-label sample speech data, then use the trained speech synthesis model to synthesize speech data with the target speech characteristics. However, this approach often relies heavily on the speech characteristics contained in the sample speech data, failing to meet actual user needs and resulting in low speech synthesis accuracy. Therefore, improving speech synthesis accuracy has become a pressing technical challenge. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, aiming to improve the accuracy of speech synthesis.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0005] Obtaining raw text data to be processed and reference speech data of a reference speaker;

[0006] Inputting the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network;

[0007] Encoding the original text data through the text encoding network to obtain an original text embedding vector;

[0008] encoding the reference speech data through the speech coding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to characterize the speech style characteristics of the reference speaker;

[0009] Aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector;

[0010] The target text embedding vector is decoupled through the decoding network to obtain synthesized speech data.

[0011] In some embodiments, encoding the original text data using the text encoding network to obtain the original text embedding vector includes:

[0012] Performing data conversion on the original text data through the text encoding network and a preset reference dictionary to obtain original phoneme data;

[0013] The original phoneme data is encoded by the text encoding network to obtain the original text embedding vector.

[0014] In some embodiments, the speech synthesis model further includes a style encoding network. Before aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector by the speech alignment network to obtain the target text embedding vector, the method includes:

[0015] Get the reference text style tag;

[0016] Inputting the reference text style label into the style encoding network, wherein the style encoding network includes a language representation layer and an activation layer;

[0017] Filtering the reference text style tags through the language representation layer to obtain a target text style tag;

[0018] The target text style label is activated by the activation layer to obtain the text style embedding vector.

[0019] In some embodiments, the speech coding network includes a first fully connected layer, a residual block, an attention layer, and a pooling layer, and encoding the reference speech data through the speech coding network to obtain a reference speech embedding vector includes:

[0020] Mapping the reference speech data to a preset vector space through the first fully connected layer to obtain a reference speech feature vector;

[0021] Performing feature extraction on the reference speech feature vector using the residual block to obtain a target speech feature vector;

[0022] Performing attention calculation on the target speech feature vector through the attention layer to obtain an initial speech embedding vector;

[0023] The initial speech embedding vector is sampled by the pooling layer to obtain the reference speech embedding vector and the reference Mel-cepstral feature.

[0024] In some embodiments, the speech alignment network includes an encoding layer and a normalization layer, and aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector includes:

[0025] Encoding the original text embedding vector through the encoding layer to obtain a text encoding vector, and encoding the reference Mel-cepstrum feature through the encoding layer to obtain a spectrum encoding vector;

[0026] Performing vector concatenation on the original text embedding vector, the reference speech embedding vector, and the text style embedding vector to obtain a preliminary text embedding vector;

[0027] Performing distance calculation on the text encoding vector and the spectrum encoding vector to obtain a characteristic distance value;

[0028] Normalizing the preliminary text embedding vector based on the activation function of the normalization layer and the feature distance value to obtain an intermediate text embedding vector;

[0029] Performing path identification on the text encoding vector and the spectrum encoding vector based on a preset algorithm to obtain target path data, and performing feature alignment on the intermediate text embedding vector according to the target path data to obtain a candidate text embedding vector;

[0030] Fine-tune the candidate text embedding vector to obtain the target text embedding vector.

[0031] In some embodiments, the speech synthesis model includes a time prediction network, and fine-tuning the candidate text embedding vector to obtain the target text embedding vector includes:

[0032] Obtaining a duration parameter, wherein the duration parameter is derived from a time prediction process of the reference speech embedding vector and the text style embedding vector by the time prediction network;

[0033] The length of the candidate text embedding vector is adjusted according to the duration parameter to obtain the target text embedding vector.

[0034] In some embodiments, the decoding network includes a prediction layer, a second fully connected layer, an LSTM layer, and a convolutional layer, and the decoupling processing of the target text embedding vector by the decoding network to obtain the synthesized speech data includes:

[0035] Performing prediction processing on the target text embedding vector through the prediction layer to obtain a predicted Mel-frequency cepstrum feature;

[0036] Activating the predicted Mel-cepstrum feature through the second fully connected layer to obtain an activated Mel-cepstrum feature;

[0037] Performing splicing processing on the activated Mel-cepstral feature, the reference speech embedding vector, and the text style embedding vector to obtain a target splicing vector;

[0038] Performing linear projection on the target splicing vector through the LSTM layer to obtain a target Mel-spectrogram frame;

[0039] Performing synthesis processing on the target Mel-spectrogram frame through the convolution layer to obtain a target Mel-spectrogram;

[0040] The target Mel spectrum is vocoded to obtain the synthesized speech data.

[0041] To achieve the above-mentioned object, a second aspect of an embodiment of the present application provides a speech synthesis device, comprising:

[0042] A data acquisition module, configured to acquire raw text data to be processed and reference speech data of a reference speaker;

[0043] An input module, configured to input the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network;

[0044] A text encoding module, configured to encode the original text data through the text encoding network to obtain an original text embedding vector;

[0045] a speech encoding module, configured to encode the reference speech data through the speech encoding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to characterize the speech style characteristics of the reference speaker;

[0046] A vector alignment module is used to align the original text embedding vector, the reference speech embedding vector and the pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector;

[0047] A decoupling module is used to decouple the target text embedding vector through the decoding network to obtain synthesized speech data.

[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0050] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application obtain original text data to be processed and reference speech data of a reference speaker; input the original text data and reference speech data into a preset speech synthesis model, which includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network; encode the original text data through the text encoding network to obtain an original text embedding vector, which can effectively improve the standardization of the text. Encode the reference speech data through the speech encoding network to obtain a reference speech embedding vector, which can enhance the extraction of important feature information in the reference speech data and improve the accuracy of the reference speech embedding vector. Align the original text embedding vector, the reference speech embedding vector, and a pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector, which can more conveniently integrate text style information and the speech characteristics of the reference speaker into the original text embedding vector, so that the target text embedding vector meets the current text style requirements and the speech characteristics of the reference speaker. Finally, the target text embedding vector is decoupled through the decoding network to obtain synthesized speech data, which can realize cross-speaker style speech synthesis, so that the synthesized speech data corresponding to the original text data contains the speech characteristics of the reference object, thereby improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of the speech synthesis method provided in an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0053] Figure 3 yes Figure 1 Flowchart of step S104 in FIG.

[0054] Figure 4 is another flow chart of the speech synthesis method provided in an embodiment of the present application;

[0055] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0056] Figure 6 yes Figure 5 Flowchart of step S506 in FIG.

[0057] Figure 7 yes Figure 1 Flowchart of step S106 in FIG.

[0058] Figure 8 is a structural diagram of a speech synthesis device provided in an embodiment of the present application;

[0059] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0061] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0063] First, let’s analyze some of the terms used in this application:

[0064] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0065] Natural language processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). A branch of artificial intelligence, NLP is an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing encompasses grammatical analysis, semantic analysis, and discourse comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It encompasses data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistics research related to language computing.

[0066] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event, and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data is all text information extraction. Of course, the information extracted by text information extraction technology can be of various types.

[0067] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct the Mel-Frequency Cepstrum. A segment of a music signal can be used to generate a cepstrum that adequately represents the signal. The Mel-Frequency Cepstrum coefficients are the cepstrum (the spectrum of the spectrum) derived from this cepstrum. Unlike conventional cepstrum, the Mel-Frequency Cepstrum's most distinctive feature is that the frequency bands on the Mel-Frequency Cepstrum are evenly distributed on the Mel scale. This means that compared to the commonly seen linear cepstrum representation, these frequency bands are more closely aligned with the nonlinear human auditory system. For example, Mel-Frequency Cepstrum is often used in audio compression techniques.

[0068] Phoneme: It is the smallest speech unit divided according to the natural properties of speech. It is analyzed based on the pronunciation actions in the syllable, and one action constitutes a phoneme.

[0069] Activation Function: It is a function that runs on the neurons of the artificial neural network and is responsible for mapping the input of the neuron to the output.

[0070] Encoder: Converts the input sequence into a fixed-length vector.

[0071] Decoder: It converts the previously generated fixed vector into an output sequence. The input sequence can be text, voice, image, or video; the output sequence can be text or image.

[0072] Speech synthesis, also known as text-to-speech (TTS), involves synthesizing understandable, natural speech from text. Common speech synthesis methods typically train models based on fixed-label sample speech data, then use the trained speech synthesis model to synthesize speech data with the target speech characteristics. However, this approach often relies heavily on the speech characteristics contained in the sample speech data, failing to meet actual user needs and resulting in low speech synthesis accuracy. Therefore, improving speech synthesis accuracy has become a pressing technical challenge.

[0073] Based on this, the embodiments of the present application provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, aiming to improve the accuracy of speech synthesis.

[0074] The speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.

[0075] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0076] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0077] The speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0079] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0081] Step S101, obtaining original text data to be processed and reference speech data of a reference speaker;

[0082] Step S102: inputting the original text data and the reference speech data into a preset speech synthesis model, where the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network;

[0083] Step S103, encoding the original text data through a text encoding network to obtain an original text embedding vector;

[0084] Step S104: encoding the reference speech data through a speech coding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to represent the speech style characteristics of the reference speaker;

[0085] Step S105, aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector through a speech alignment network to obtain a target text embedding vector;

[0086] Step S106: Decoupling the target text embedding vector through a decoding network to obtain synthesized speech data.

[0087] In the embodiment of the present application, steps S101 to S106 are performed by obtaining the original text data to be processed and the reference speech data of the reference speaker; inputting the original text data and the reference speech data into a preset speech synthesis model, which includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network; encoding the original text data through the text encoding network to obtain the original text embedding vector, which can effectively improve the standardization of the text. Encoding the reference speech data through the speech encoding network to obtain the reference speech embedding vector can enhance the extraction of important feature information in the reference speech data and improve the accuracy of the reference speech embedding vector. Aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector through the speech alignment network to obtain the target text embedding vector can more conveniently integrate the text style information and the speech characteristics of the reference object into the original text embedding vector, so that the target text embedding vector meets the current text style requirements and the speech characteristics of the reference object. Finally, the target text embedding vector is decoupled through the decoding network to obtain synthesized speech data, which can realize cross-speaker style speech synthesis, so that the synthesized speech data corresponding to the original text data contains the speech characteristics of the reference object, thereby improving the accuracy of speech synthesis.

[0088] In step S101 of some embodiments, raw text data can be obtained from a public dataset, or from an existing text database or online platform, without limitation. For example, the public dataset can be the LJSpeech dataset, etc. Furthermore, a web crawler can be written, and after setting up a data source, targeted data crawling can be performed to obtain reference speech data of a reference speaker. The data source can be various types of online platforms, social media, or certain specific audio databases, and the reference speech data can be music materials, speech reports, chat conversations, etc. of the reference speaker. This method makes it relatively easy to obtain reference speech data and raw text data, improving data acquisition efficiency.

[0089] In step S102 of some embodiments, the original text data and reference speech data can be input into a preset speech synthesis model through a preset script program or other computer program, wherein the speech synthesis model can be constructed based on the Tacotron2 network, and the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network and a decoding network. The text encoding network is mainly used to encode the input original text data, extract the phoneme features of the original text data, and obtain the original text embedding vector. The speech encoding network is mainly used to encode the reference speech data, extract the spectral features of the reference speech data and the speech characteristics of the reference object, and obtain the Mel-frequency cepstral features representing the speech content and the reference speech embedding features representing the speech characteristics of the reference object. The speech alignment network is mainly used to align the original text embedding vector and the Mel-frequency cepstral features based on the reference speech embedding features, so that the generated target text embedding vector has better coherence. The decoding network is mainly used to decode the target text embedding vector, obtain the corresponding target Mel-frequency spectrum, and perform voice coding on the target Mel-frequency spectrum to obtain the synthesized speech data corresponding to the original text data. This speech synthesis model can better achieve the matching alignment between Mel-frequency cepstral features and original text data, and embed the preset text style information into the synthesized speech data, so that the synthesized speech data can contain the required emotional and rhythmic features, thereby improving the accuracy of speech synthesis.

[0090] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:

[0091] Step S201, performing data conversion on original text data through a text encoding network and a preset reference dictionary to obtain original phoneme data;

[0092] Step S202: Encode the original phoneme data through a text encoding network to obtain an original text embedding vector.

[0093] In some embodiments, in step S201, the preset reference dictionary may be a CMUSphinx data dictionary, which includes multiple words. A word list may be constructed using the text encoding network and the reference dictionary. The word list is formatted as one word or character per line. This word list is used to perform data conversion on the text content in the original text data, converting the words or characters corresponding to the text content into phoneme sequences, thereby forming original phoneme data corresponding to the original text data.

[0094] In step S202 of some embodiments, the original phoneme data is encoded through a text encoding network, and the original phoneme data, which is a phoneme sequence in text form, is converted into a vector form to obtain an original text embedding vector.

[0095] By performing format conversion and data conversion on the original text data through the above steps S201 to S202, the standardization of the text can be effectively improved, and the phoneme features in the original text data can be more conveniently extracted, and the original text data can be processed into an original text embedding vector in vector form.

[0096] See also Figure 3 In some embodiments, the speech coding network includes a first fully connected layer, a residual block, an attention layer, and a pooling layer. Step S104 may include but is not limited to steps S301 to S304:

[0097] Step S301, mapping the reference speech data to a preset vector space through a first fully connected layer to obtain a reference speech feature vector;

[0098] Step S302, extracting features from the reference speech feature vector using a residual block to obtain a target speech feature vector;

[0099] Step S303, performing attention calculation on the target speech feature vector through the attention layer to obtain an initial speech embedding vector;

[0100] Step S304: Sampling the initial speech embedding vector through a pooling layer to obtain a reference speech embedding vector and a reference Mel-frequency cepstrum feature.

[0101] In step S301 of some embodiments, the reference speech data is mapped to a preset vector space through a first fully connected layer, thereby converting the reference speech data from a spectrum space to a vector space and obtaining a reference speech feature vector.

[0102] In step S302 of some embodiments, feature extraction is performed on the reference speech feature vector using a residual block to capture the speech content and characteristics of the reference speech feature vector as comprehensively as possible, thereby obtaining a target speech feature vector. Furthermore, using a residual block for feature extraction can effectively address the vanishing gradient issue during feature extraction and improve the stability of feature extraction.

[0103] In step S303 of some embodiments, when the attention layer performs attention calculation on the target speech feature vector, the key matrix, value matrix and query matrix of the target speech feature vector are calculated, and the key matrix, value matrix and query matrix are weighted by the softmax function to obtain the initial speech embedding vector, wherein the key matrix can be expressed as K1=X1*W1, the value matrix can be expressed as V1=X1*W2, and the query matrix can be expressed as Q1=X1*W3, wherein X1 is the target speech feature vector, and W1, W2, and W3 are trainable parameters. The initial speech embedding vector Z1 can be expressed as shown in formula (1), wherein d1 is the feature dimension of the target speech feature vector, and T represents the transposition operation of the key matrix K1:

[0104]

[0105] The above attention calculation process can strengthen the mapping of important feature information in the target speech feature vector, while reducing the mapping of secondary feature information in the first intermediate speech feature.

[0106] In step S304 of some embodiments, the initial speech embedding vector is downsampled through a pooling layer. The downsampling process can be either maximum pooling or average pooling. In this way, redundant information of the initial speech embedding vector can be better eliminated, and more accurate speech content and speech characteristics of the reference object can be obtained, thereby obtaining a reference speech embedding vector representing the speech style characteristics of the reference speaking object and a reference Mel-frequency cepstral feature representing the speech content, wherein the speech style characteristics include pitch, timbre, vocalization frequency, etc.

[0107] Through the above steps S301 to S304, the extraction of important feature information in the reference speech data can be strengthened, and the reference Mel-frequency cepstral features and reference speech embedding vectors for speech synthesis can be obtained, so that the subsequently generated synthesized speech data can contain more speech style characteristics of the reference speaker, thereby improving the accuracy of speech synthesis.

[0108] See also Figure 4 In some embodiments, before step S105, the speech synthesis model further includes a style encoding network. The speech synthesis method may include but is not limited to steps S401 to S404:

[0109] Step S401, obtaining a reference text style tag;

[0110] Step S402: inputting the reference text style label into a style encoding network, which includes a language representation layer and an activation layer;

[0111] Step S403, filtering the reference text style tags through the language representation layer to obtain the target text style tags;

[0112] Step S404: Activate the target text style label through the activation layer to obtain a text style embedding vector.

[0113] In step S401 of some embodiments, a reference text style tag may be obtained from a preset tag library or other data platform, wherein the reference text style tag is a brief description of the speaking style, including information about emotion, intention, tone, speed, etc.

[0114] In step S402 of some embodiments, the reference text style label can be input into the style encoding network through a preset script program or other computer program, wherein the style encoding network includes a language representation layer and an activation layer. The language representation layer is mainly used to filter and process the reference text style label according to the current speech synthesis requirements, and the activation layer is mainly used to map the target text style label obtained after the filtering process to a preset vector space to obtain a text style embedding vector.

[0115] In step S403 of some embodiments, when the reference text style tags are screened by the language representation layer, the language representation layer can screen the reference text style tags based on the reference speech embedding vector, calculate the similarity between the reference speech embedding vector and each reference text style tag by using a cosine similarity algorithm, and select the reference text style tag with the highest similarity as the target text style tag.

[0116] In step S404 of some embodiments, the target text style tag is activated using an activation function of the activation layer, mapping the target text style tag to a preset vector space to obtain a text style embedding vector. The activation function may be a Rule function, etc., without limitation.

[0117] Through the above steps S401 to S404, a label that meets the current speech synthesis requirements, namely the target text style label, can be screened out from multiple reference text style labels, and the target text style label can be converted into a text style embedding vector that can assist in speech synthesis, so that in subsequent speech synthesis, the text style information can be embedded into the synthesized speech data, thereby improving the accuracy of speech synthesis.

[0118] See also Figure 5In some embodiments, the speech alignment network includes an encoding layer and a normalization layer. Step S105 may include but is not limited to steps S501 to S506:

[0119] Step S501: encoding the original text embedding vector through the encoding layer to obtain a text encoding vector, and encoding the reference Mel-cepstrum feature through the encoding layer to obtain a spectrum encoding vector;

[0120] Step S502: performing vector concatenation on the original text embedding vector, the reference speech embedding vector, and the text style embedding vector to obtain a preliminary text embedding vector;

[0121] Step S503, performing distance calculation on the text encoding vector and the spectrum encoding vector to obtain a characteristic distance value;

[0122] Step S504, normalizing the preliminary text embedding vector based on the activation function of the normalization layer and the feature distance value to obtain an intermediate text embedding vector;

[0123] Step S505: performing path identification on the text encoding vector and the spectrum encoding vector based on a preset algorithm to obtain target path data, and performing feature alignment on the intermediate text embedding vector according to the target path data to obtain a candidate text embedding vector;

[0124] Step S506: Fine-tune the candidate text embedding vector to obtain the target text embedding vector.

[0125] In step S501 of some embodiments, the original text embedding vector is encoded through the encoding layer, and the feature dimension of the original text embedding vector is adjusted to obtain a text encoding vector, and the reference Mel-cepstral feature is encoded through the encoding layer, and the feature dimension of the reference Mel-cepstral feature is adjusted to obtain a spectrum encoding vector. In this way, the spectrum encoding vector and the text encoding vector are in the same feature dimension.

[0126] In step S502 of some embodiments, vector concatenation is performed on the original text embedding vector, the reference speech embedding vector, and the text style embedding vector to obtain a vector with a longer vector length, namely, the preliminary text embedding vector.

[0127] In step S503 of some embodiments, the distance between the text encoding vector and the spectrum encoding vector is calculated using the Euclidean distance or Manhattan distance calculation method, and the calculated Euclidean distance or Manhattan distance is used as the feature distance value.

[0128] In step S504 of some embodiments, the activation function includes a softmax function, etc., and the initial text embedding vector is fixed to a preset vector region through the activation function of the normalization layer, and feature alignment of the initial text embedding vector is performed in the vector region based on the feature distance value. Specifically, the feature distance values ​​of each phoneme feature in the text encoding vector and the spectrum encoding vector are averaged to obtain a distance mean, and the feature distance between each phoneme feature of the initial text embedding vector and the spectrum encoding vector in the vector region is adjusted to the distance mean to obtain the intermediate text embedding vector.

[0129] In step S505 of some embodiments, the preset algorithms include shortest path algorithms such as Dijkstra algorithm, Bellman-Ford algorithm, Floyd algorithm and SPFA algorithm. The text encoding vector and the spectrum encoding vector are identified by the shortest path algorithm, the shortest path between the text encoding vector and the spectrum encoding vector is obtained, and the target path data is obtained. The intermediate text embedding vector is feature aligned according to the shortest path, so that the path between the intermediate text embedding vector and the spectrum encoding vector is consistent with the shortest path, and the candidate text embedding vector is obtained.

[0130] In step S506 of some embodiments, since the speaking styles of different reference objects will affect the duration of the factors, it is necessary to adjust the candidate text embedding vector according to the speech characteristics of different reference objects. First, it is necessary to perform time prediction processing on the reference speech embedding vector and the text style embedding vector based on the time prediction network of the speech synthesis model to obtain a duration parameter, which can characterize the number of frames of each phoneme. This prediction process may include operations such as convolution processing and normalization processing on the reference speech embedding vector and the text style embedding vector. Then, the length of the candidate text embedding vector is adjusted according to the duration parameter to obtain a target text embedding vector, so that the length of the target text embedding vector is basically consistent with the length of the reference speech embedding vector.

[0131] Through the above steps S501 to S506, the text style information and the voice characteristics of the reference object can be more conveniently integrated into the original text embedding vector, so that the target text embedding vector meets the current text style requirements and the voice characteristics of the reference object.

[0132] See also Figure 6 In some embodiments, the speech synthesis model includes a time prediction network, and step S506 includes but is not limited to steps S601 to S602:

[0133] Step S601, obtaining a duration parameter, wherein the duration parameter is derived from a time prediction process of a reference speech embedding vector and a text style embedding vector by a time prediction network;

[0134] Step S602: Adjust the length of the candidate text embedding vector according to the duration parameter to obtain the target text embedding vector.

[0135] In step S601 of some embodiments, time prediction processing is first performed on the reference speech embedding vector and the text style embedding vector based on the time prediction network of the speech synthesis model. This process includes convolution processing, normalization processing and other operations on the reference speech embedding vector and the text style embedding vector.

[0136] In step S602 of some embodiments, the length of the candidate text embedding vector is adjusted according to the duration parameter to obtain a target text embedding vector, so that the length of the target text embedding vector is substantially consistent with the length of the reference speech embedding vector.

[0137] Through the above steps S601 to S602, the duration of the phonemes in the target text embedding vector can be made consistent with the speech characteristics of the reference object, thereby improving the accuracy of speech synthesis.

[0138] See also Figure 7 In some embodiments, the decoding network includes a prediction layer, a second fully connected layer, an LSTM layer, and a convolutional layer. Step S106 may include but is not limited to steps S701 to S706:

[0139] Step S701, predicting the target text embedding vector through the prediction layer to obtain predicted Mel-cepstrum features;

[0140] Step S702, performing activation processing on the predicted Mel-cepstrum feature through the second fully connected layer to obtain an activated Mel-cepstrum feature;

[0141] Step S703, concatenating the activated Mel-cepstral feature, the reference speech embedding vector, and the text style embedding vector to obtain a target concatenated vector;

[0142] Step S704: linearly project the target concatenated vector through the LSTM layer to obtain a target mel-spectrogram frame;

[0143] Step S705, synthesizing the target mel-spectrogram frame through a convolutional layer to obtain a target mel-spectrogram;

[0144] Step S706: Perform vocoding on the target Mel-spectrogram to obtain synthesized speech data.

[0145] In step S701 of some embodiments, the target text embedding vector is decoded by a prediction layer to capture the phoneme feature information in the target text embedding vector and output a predicted Mel-cepstral feature.

[0146] In step S702 of some embodiments, when the predicted Mel-cepstral feature is activated by the second fully connected layer, the predicted Mel-cepstral feature can be moved to a preset activation space, and then the predicted Mel-cepstral feature in the activation area is activated by the ReLU function to obtain the activated Mel-cepstral feature.

[0147] In step S703 of some embodiments, the activated Mel-cepstral features, the reference speech embedding vector, and the text style embedding vector can be first processed to the same vector dimension, and then the activated Mel-cepstral features, the reference speech embedding vector, and the text style embedding vector in the same vector dimension are processed and vector added or vector spliced ​​to obtain a target spliced ​​vector.

[0148] In step S704 of some embodiments, a linear projection is performed on the target splicing vector through an LSTM layer, and the target splicing vector is projected into a preset frequency domain space, thereby predicting a target Mel-ceptrogram frame corresponding to the synthesized speech data.

[0149] In step S705 of some embodiments, the target mel-spectrogram frames are synthesized using a convolutional layer, and each target mel-spectrogram frame is sequentially concatenated to obtain a target mel-spectrogram. The concatenation order can be determined based on the context order of the original text data or by other methods, without limitation.

[0150] In step S706 of some embodiments, the target Mel spectrum may be vocoded by a vocoder to convert the target Mel spectrum in spectrum form into audio data in waveform form to obtain synthesized speech data.

[0151] Through the decoding processing and linear projection of the above steps S701 to S706, synthesized speech data can be easily generated to achieve cross-speaker style speech synthesis, so that the synthesized speech data corresponding to the original text data contains the speech characteristics of the reference object, thereby improving the accuracy of speech synthesis.

[0152] The speech synthesis method of the present invention obtains raw text data to be processed and reference speech data of a reference speaker; inputs the raw text data and reference speech data into a preset speech synthesis model, which includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network; encodes the raw text data through the text encoding network to obtain a raw text embedding vector, which can effectively improve the standardization of the text; encodes the reference speech data through the speech encoding network to obtain a reference speech embedding vector, which can enhance the extraction of important feature information from the reference speech data and improve the accuracy of the reference speech embedding vector; aligns the raw text embedding vector, the reference speech embedding vector, and a pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector, which can more conveniently incorporate text style information and the speech characteristics of the reference speaker into the raw text embedding vector, so that the target text embedding vector meets the current text style requirements and the speech characteristics of the reference speaker. Finally, decouples the target text embedding vector through the decoding network to obtain synthesized speech data, which can achieve cross-speaker speech synthesis, so that the synthesized speech data corresponding to the raw text data contains the speech characteristics of the reference speaker, thereby improving the accuracy of speech synthesis.

[0153] See also Figure 8 The present application also provides a speech synthesis device that can implement the above-mentioned speech synthesis method. The device includes:

[0154] The data acquisition module 801 is used to acquire the original text data to be processed and the reference speech data of the reference speaker;

[0155] An input module 802 is configured to input the original text data and reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network;

[0156] A text encoding module 803 is configured to encode the original text data through a text encoding network to obtain an original text embedding vector;

[0157] The speech encoding module 804 is configured to encode the reference speech data through a speech encoding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to represent the speech style characteristics of the reference speaker;

[0158] A vector alignment module 805 is configured to align the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector using a speech alignment network to obtain a target text embedding vector;

[0159] The decoupling module 806 is used to decouple the target text embedding vector through a decoding network to obtain synthesized speech data.

[0160] In some embodiments, the text encoding module 803 includes:

[0161] A data conversion unit, configured to convert the original text data using a text encoding network and a preset reference dictionary to obtain original phoneme data;

[0162] The encoding unit is used to encode the original phoneme data through the text encoding network to obtain the original text embedding vector.

[0163] In some embodiments, the speech coding network includes a first fully connected layer, a residual block, an attention layer, and a pooling layer, and the speech coding module 804 includes:

[0164] A mapping unit, configured to map the reference speech data to a preset vector space through a first fully connected layer to obtain a reference speech feature vector;

[0165] A feature extraction unit, configured to extract features from a reference speech feature vector using a residual block to obtain a target speech feature vector;

[0166] An attention calculation unit is used to perform attention calculation on the target speech feature vector through the attention layer to obtain an initial speech embedding vector;

[0167] The sampling unit is used to sample the initial speech embedding vector through the pooling layer to obtain a reference speech embedding vector and a reference Mel-frequency cepstrum feature.

[0168] In some embodiments, the speech synthesis model further includes a style encoding network, and the speech synthesis method further includes a vector acquisition module, specifically including:

[0169] A label acquisition unit, used to acquire a reference text style label;

[0170] A label input unit, used to input the reference text style label into the style encoding network, which includes a language representation layer and an activation layer;

[0171] A screening unit, configured to screen the reference text style labels through a language representation layer to obtain a target text style label;

[0172] The activation unit is used to activate the target text style label through the activation layer to obtain the text style embedding vector.

[0173] In some embodiments, the speech alignment network includes an encoding layer and a normalization layer, and the vector alignment module 805 includes:

[0174] A feature encoding unit is used to encode the original text embedding vector through the encoding layer to obtain a text encoding vector, and to encode the reference Mel-frequency cepstrum feature through the encoding layer to obtain a spectrum encoding vector;

[0175] A vector connection unit is used to perform vector connection on the original text embedding vector, the reference speech embedding vector, and the text style embedding vector to obtain a preliminary text embedding vector;

[0176] A distance calculation unit is used to calculate the distance between the text encoding vector and the spectrum encoding vector to obtain a characteristic distance value;

[0177] A vector normalization unit is used to normalize the preliminary text embedding vector based on the activation function and feature distance value of the normalization layer to obtain an intermediate text embedding vector;

[0178] A feature alignment unit is used to perform path identification on the text encoding vector and the spectrum encoding vector based on a preset algorithm to obtain target path data, and perform feature alignment on the intermediate text embedding vector according to the target path data to obtain a candidate text embedding vector;

[0179] The fine-tuning unit is used to fine-tune the candidate text embedding vector to obtain the target text embedding vector.

[0180] In some embodiments, the speech synthesis model includes a temporal prediction network, and the fine-tuning unit includes:

[0181] a parameter acquisition subunit, configured to acquire a duration parameter, wherein the duration parameter is derived from a time prediction process of a reference speech embedding vector and a text style embedding vector by a time prediction network;

[0182] The adjustment subunit is used to adjust the length of the candidate text embedding vector according to the duration parameter to obtain the target text embedding vector.

[0183] In some embodiments, the decoding network includes a prediction layer, a second fully connected layer, an LSTM layer, and a convolutional layer, and the decoupling module 806 includes:

[0184] The prediction unit is used to predict the target text embedding vector through the prediction layer to obtain the predicted Mel-frequency cepstrum feature;

[0185] A feature activation unit is used to activate the predicted Mel-cepstrum feature through the second fully connected layer to obtain an activated Mel-cepstrum feature;

[0186] A concatenation unit is used to concatenate the activated Mel-frequency cepstral features, the reference speech embedding vector, and the text style embedding vector to obtain a target concatenation vector.

[0187] The projection unit is used to linearly project the target concatenated vector through the LSTM layer to obtain the target Mel-spectrogram frame;

[0188] A synthesis unit is used to synthesize the target Mel spectrum frame through a convolution layer to obtain a target Mel spectrum;

[0189] The conversion unit is used to perform voice coding conversion on the target Mel spectrum to obtain synthesized speech data.

[0190] The specific implementation of the speech synthesis device is basically the same as the specific embodiment of the above-mentioned speech synthesis method, and will not be repeated here.

[0191] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned speech synthesis method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0192] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0193] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0194] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the speech synthesis method of the embodiments of this application.

[0195] Input / output interface 903, used to implement information input and output;

[0196] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0197] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0198] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0199] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech synthesis method is implemented.

[0200] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0201] The speech synthesis method, speech synthesis device, electronic device and computer-readable storage medium provided in the embodiments of the present application obtain the original text data to be processed and the reference speech data of the reference speaking object; input the original text data and the reference speech data into a preset speech synthesis model, the speech synthesis model including a text encoding network, a speech encoding network, a speech alignment network and a decoding network; encode the original text data through the text encoding network to obtain the original text embedding vector, which can effectively improve the standardization of the text. Encode the reference speech data through the speech encoding network to obtain the reference speech embedding vector, which can enhance the extraction of important feature information in the reference speech data and improve the accuracy of the reference speech embedding vector. Align the original text embedding vector, the reference speech embedding vector and the pre-acquired text style embedding vector through the speech alignment network to obtain the target text embedding vector, which can more conveniently integrate the text style information and the speech characteristics of the reference object into the original text embedding vector, so that the target text embedding vector meets the current text style requirements and the speech characteristics of the reference object. Finally, the target text embedding vector is decoupled through the decoding network to obtain synthesized speech data, which can realize cross-speaker style speech synthesis, so that the synthesized speech data corresponding to the original text data contains the speech characteristics of the reference object, thereby improving the accuracy of speech synthesis.

[0202] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0203] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0205] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0206] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0207] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0212] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtaining raw text data to be processed and reference speech data of a reference speaker; Inputting the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network; The original text data is encoded by the text encoding network to obtain an original text embedding vector; the reference speech data is encoded by the speech encoding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to represent the speech style characteristics of the reference speaker; The original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector are aligned by the speech alignment network to obtain a target text embedding vector; the target text embedding vector is decoupled by the decoding network to obtain synthesized speech data; The speech alignment network includes an encoding layer and a normalization layer. The speech alignment network aligns the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector to obtain a target text embedding vector, including: The original text embedding vector is encoded by the encoding layer to obtain a text encoding vector, and the reference Mel-cepstral feature is encoded by the encoding layer to obtain a spectrum encoding vector; the original text embedding vector, the reference speech embedding vector and the text style embedding vector are vector-connected to obtain a preliminary text embedding vector; distance calculation is performed on the text encoding vector and the spectrum encoding vector to obtain a feature distance value; the preliminary text embedding vector is normalized based on the activation function of the normalization layer and the feature distance value to obtain an intermediate text embedding vector; path identification is performed on the text encoding vector and the spectrum encoding vector based on a preset algorithm to obtain target path data, and feature alignment is performed on the intermediate text embedding vector according to the target path data to obtain a candidate text embedding vector; the candidate text embedding vector is fine-tuned to obtain the target text embedding vector, wherein the reference Mel-cepstral feature is obtained by encoding the original text data through the text encoding network.

2. The method according to claim 1, characterized in that The encoding process of the original text data by the text encoding network to obtain the original text embedding vector includes: Performing data conversion on the original text data through the text encoding network and a preset reference dictionary to obtain original phoneme data; The original phoneme data is encoded by the text encoding network to obtain the original text embedding vector.

3. The method according to claim 1, characterized in that The speech synthesis model further includes a style encoding network. Before aligning the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector by the speech alignment network to obtain the target text embedding vector, the method includes: Get the reference text style tag; Inputting the reference text style label into the style encoding network, wherein the style encoding network includes a language representation layer and an activation layer; Filtering the reference text style tags through the language representation layer to obtain a target text style tag; The target text style label is activated by the activation layer to obtain the text style embedding vector.

4. The method according to claim 1, wherein The speech coding network includes a first fully connected layer, a residual block, an attention layer, and a pooling layer. The speech coding network encodes the reference speech data to obtain a reference speech embedding vector, including: Mapping the reference speech data to a preset vector space through the first fully connected layer to obtain a reference speech feature vector; Performing feature extraction on the reference speech feature vector using the residual block to obtain a target speech feature vector; Performing attention calculation on the target speech feature vector through the attention layer to obtain an initial speech embedding vector; The initial speech embedding vector is sampled by the pooling layer to obtain the reference speech embedding vector and the reference Mel-cepstral feature.

5. The method according to claim 1, wherein The speech synthesis model includes a time prediction network, and the fine-tuning of the candidate text embedding vector to obtain the target text embedding vector includes: Obtaining a duration parameter, wherein the duration parameter is derived from a time prediction process of the reference speech embedding vector and the text style embedding vector by the time prediction network; The length of the candidate text embedding vector is adjusted according to the duration parameter to obtain the target text embedding vector.

6. The method according to any one of claims 1 to 5, characterized in that The decoding network includes a prediction layer, a second fully connected layer, an LSTM layer, and a convolutional layer. The target text embedding vector is decoupled by the decoding network to obtain synthesized speech data, including: Performing prediction processing on the target text embedding vector through the prediction layer to obtain a predicted Mel-frequency cepstrum feature; Activating the predicted Mel-cepstrum feature through the second fully connected layer to obtain an activated Mel-cepstrum feature; Performing splicing processing on the activated Mel-cepstral feature, the reference speech embedding vector, and the text style embedding vector to obtain a target splicing vector; Performing linear projection on the target splicing vector through the LSTM layer to obtain a target Mel-spectrogram frame; Performing synthesis processing on the target Mel-spectrogram frame through the convolution layer to obtain a target Mel-spectrogram; The target Mel spectrum is vocoded to obtain the synthesized speech data.

7. A speech synthesis device, characterized in that: The device comprises: A data acquisition module, configured to acquire raw text data to be processed and reference speech data of a reference speaker; An input module, configured to input the original text data and the reference speech data into a preset speech synthesis model, wherein the speech synthesis model includes a text encoding network, a speech encoding network, a speech alignment network, and a decoding network; A text encoding module, configured to encode the original text data through the text encoding network to obtain an original text embedding vector; a speech encoding module, configured to encode the reference speech data through the speech encoding network to obtain a reference speech embedding vector, wherein the reference speech embedding vector is used to characterize the speech style characteristics of the reference speaker; A vector alignment module is used to align the original text embedding vector, the reference speech embedding vector and the pre-acquired text style embedding vector through the speech alignment network to obtain a target text embedding vector; A decoupling module, configured to perform decoupling processing on the target text embedding vector through the decoding network to obtain synthesized speech data; The speech alignment network includes an encoding layer and a normalization layer. The speech alignment network aligns the original text embedding vector, the reference speech embedding vector, and the pre-acquired text style embedding vector to obtain a target text embedding vector, including: The original text embedding vector is encoded by the encoding layer to obtain a text encoding vector, and the reference Mel-cepstral feature is encoded by the encoding layer to obtain a spectrum encoding vector; the original text embedding vector, the reference speech embedding vector and the text style embedding vector are vector-connected to obtain a preliminary text embedding vector; distance calculation is performed on the text encoding vector and the spectrum encoding vector to obtain a feature distance value; the preliminary text embedding vector is normalized based on the activation function of the normalization layer and the feature distance value to obtain an intermediate text embedding vector; path identification is performed on the text encoding vector and the spectrum encoding vector based on a preset algorithm to obtain target path data, and feature alignment is performed on the intermediate text embedding vector according to the target path data to obtain a candidate text embedding vector; the candidate text embedding vector is fine-tuned to obtain the target text embedding vector, wherein the reference Mel-cepstral feature is obtained by encoding the original text data through the text encoding network.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images