Speech synthesis method and device, electronic equipment and storage medium
By training the target speech feature extraction sub-model and text coding sub-model, combined with the audio signal generation sub-model, the problem of limitations in the existing speech synthesis methods is solved, and efficient and diverse speech synthesis is achieved.
Patent Information
- Application Number
- CN202411693142.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-28
AI Technical Summary
The synthetic speech characteristics generated by existing speech synthesis methods have limitations, and it is difficult to improve the diversity of synthetic speech characteristics.
By obtaining speech synthesis data and speech synthesis model, the target speech feature extraction sub-model is trained, the target audio feature vector is extracted, and the vector map is combined with text encoding vectors to generate speech encoding vectors, and finally speech synthesis is performed through the audio signal generation sub-model.
It improves the nature and diversity of synthetic pronunciations, ensures that the generated pronunciations can reflect rich emotions and intonations, and enhances the diversity of pronunciation.
Smart Images
Figure CN119541454A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, applicable to the field of financial technology, and in particular to a speech synthesis method and device, electronic equipment and storage medium. Background Art
[0002] Speech synthesis is an artificial intelligence technology that converts text information into speech. It can enable computers to "speak" by simulating or synthesizing human voices. For example, in the financial field, by converting product promotion text in financial documents into speech, users who have difficulty reading text can obtain product promotion information through hearing.
[0003] At present, the common speech synthesis method is to use a trained speech synthesis model to convert text information into speech to obtain synthesized speech. However, the synthesized speech generated by this method often only has the characteristics of the speech samples used in model training, which limits the speech characteristics of the synthesized speech. Therefore, how to improve the diversity of synthesized speech features has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis method and device, an electronic device and a storage medium, aiming to improve the diversity of synthesized speech features.
[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:
[0006] Acquire speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data;
[0007] Acquire a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model;
[0008] Acquire speech sample data, and train the initial speech feature extraction sub-model based on the speech sample data to obtain a target speech feature extraction sub-model;
[0009] Based on the target speech feature extraction sub-model, feature extraction is performed on the target audio data to obtain a target audio feature vector;
[0010] Based on the text encoding sub-model, text encoding is performed on the speech synthesis text to obtain a text encoding vector;
[0011] Based on the target audio feature vector, performing vector mapping on the text encoding vector to obtain a speech encoding vector;
[0012] A sub-model is generated based on the audio signal to perform speech synthesis on the speech coding vector.
[0013] In some embodiments, the generating a sub-model based on the audio signal and performing speech synthesis on the speech coding vector to obtain synthesized speech data includes:
[0014] Based on the audio signal generating sub-model, feature extraction is performed on the speech coding vector to obtain acoustic features;
[0015] Based on the acoustic features, generating sound waves for the speech coding vector to obtain target audio sound waves;
[0016] Convert the target audio sound wave into a digital audio format.
[0017] The digital audio is data packaged to obtain the synthesized speech data.
[0018] In some embodiments, the step of extracting features from the target audio data based on the target speech feature extraction submodel to obtain a target audio feature vector includes:
[0019] Based on the target speech feature extraction sub-model, encoding the target audio data to obtain a target audio encoding vector;
[0020] Based on the target speech feature extraction sub-model, the target audio coding vector is quantized to obtain the target audio feature vector.
[0021] In some embodiments, encoding the target audio data based on the target speech feature extraction sub-model to obtain a target audio encoding vector includes:
[0022] Based on the target speech feature extraction sub-model, feature recognition is performed on the target audio data to obtain a target audio feature representation;
[0023] Capturing dynamic features of the target audio feature representation to obtain target audio dynamic features;
[0024] Based on the target audio dynamic features, feature optimization is performed on the target audio feature vector representation to obtain the target audio encoding vector.
[0025] In some embodiments, the step of performing quantization processing on the target audio coding vector based on the target speech feature extraction sub-model to obtain the target audio feature vector includes:
[0026] Based on the target speech feature extraction sub-model, the target audio coding vector is discretely mapped to obtain a discrete codeword, wherein the discrete codeword is a discrete representation of the target audio coding vector;
[0027] Based on the discrete codeword, a feature query is performed on a preset codeword feature mapping table to obtain a discrete feature, wherein the codeword feature mapping table is used to represent a matching relationship between the discrete codeword and the discrete feature;
[0028] The discrete features are combined to obtain the target audio feature vector.
[0029] In some embodiments, performing vector mapping on the text encoding vector based on the target audio feature vector to obtain a speech encoding vector includes:
[0030] Performing vector concatenation of the target audio feature vector and the text encoding vector to obtain a text feature concatenation vector;
[0031] The text feature concatenation vector is subjected to nonlinear transformation to obtain the speech coding vector.
[0032] In some embodiments, the text encoding the speech synthesis text based on the text encoding sub-model to obtain a text encoding vector includes:
[0033] Performing text standardization processing on the speech synthesis text to obtain a converted text;
[0034] Performing word segmentation processing on the converted text to obtain speech synthesis phrases;
[0035] Based on the text encoding sub-model, encoding the speech synthesis phrase to obtain a phrase encoding vector;
[0036] Based on the text encoding sub-model, the phrase encoding vectors are merged to obtain the text encoding vector.
[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising:
[0038] A data acquisition module, used to acquire speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data;
[0039] A model acquisition module, used to acquire a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model;
[0040] A model training module, used for acquiring speech sample data, and training the initial speech feature extraction sub-model based on the speech sample data to obtain a target speech feature extraction sub-model;
[0041] A feature extraction module, used to extract features from the target audio data based on the target speech feature extraction sub-model to obtain a target audio feature vector;
[0042] A text encoding module, used for performing text encoding on the speech synthesis text based on the text encoding sub-model to obtain a text encoding vector;
[0043] A speech coding module, used for performing vector mapping on the text coding vector based on the target audio feature vector to obtain a speech coding vector;
[0044] The speech synthesis module is used to generate a sub-model based on the audio signal, perform speech synthesis on the speech coding vector, and obtain synthesized speech data.
[0045] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0047] The speech synthesis method and device, electronic device and storage medium proposed in the present application obtain speech synthesis data including speech synthesis text and target audio data, obtain a speech synthesis model including an initial speech feature extraction submodel, a text encoding submodel and an audio signal generation submodel, obtain speech sample data, and train the initial speech feature extraction submodel based on the speech sample data to obtain a target speech feature extraction submodel, so as to facilitate the subsequent efficient acquisition of the target audio feature vector and ensure the naturalness and diversity of the generated speech. Secondly, according to the target speech feature extraction submodel, feature extraction is performed on the target audio data to obtain the target audio feature vector. According to the text encoding submodel, text encoding is performed on the speech synthesis text to obtain the text encoding vector. According to the target audio feature vector, vector mapping is performed on the text encoding vector to obtain the speech encoding vector, so as to improve the accuracy of synthesis and ensure that the generated speech can reflect rich emotions and intonation. Finally, according to the audio signal generation submodel, speech synthesis is performed on the speech encoding vector, so as to realize the generation of new feature speech data with zero samples and enhance the diversity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of a speech synthesis method provided in an embodiment of the present application;
[0049] Figure 2 yes Figure 1 Flow chart of step S104 in FIG.
[0050] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.
[0051] Figure 4 yes Figure 2 Flow chart of step S202 in FIG.
[0052] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.
[0053] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.
[0054] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.
[0055] Figure 8 is a structural schematic diagram of a speech synthesis device provided in an embodiment of the present application;
[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0060] First, some nouns involved in this application are analyzed:
[0061] GPT (Generative Pre-trained Transformer) model: The GPT model is a natural language processing model based on the transformer architecture. It learns the statistical characteristics and semantic information of the language by pre-training on a large amount of text data. The GPT model can be used for various generative tasks, such as text generation, language translation, and text summarization. Its powerful language understanding ability makes it an important application in the field of natural language generation.
[0062] LM (Language Model) head: The LM head is the output layer of the transformer model, which is used to predict the probability distribution of the next word. In the GPT model, the LM head linearly projects the output vector of the transformer and applies an activation function to obtain the probability of each word to generate text. The LM head is a key part of implementing the language model function.
[0063] VQ-VAE (Vector Quantized-Variational Autoencoder): VQ-VAE is a generative model that combines variational autoencoders and vector quantization techniques. It improves the quality and diversity of generated samples by learning discrete latent representations of data. VQ-VAE retains important features of the data in the latent space while optimizing the maximum likelihood, making it a powerful unsupervised representation learning model.
[0064] Decoder: The decoder is responsible for converting the encoded text features or speech features back into audible audio signals in the speech synthesis system. It receives the encoding vector from the text encoding submodel or the acoustic features from the vocoder, then generates the corresponding audio waveform and finally outputs the synthesized speech data.
[0065] Speaker encoder: The speaker encoder is used to extract speaker features from audio samples, such as timbre, pitch, etc. These features can represent the speaker's unique voice attributes, allowing the speech synthesis system to imitate the voice characteristics of a specific speaker and achieve personalized speech output.
[0066] Perceptron Conditioner: The perceptron conditioner is used to further process the features extracted by the speaker encoder to better capture the dynamic characteristics of the speaker. This involves using autoencoders or other generative models to improve the feature representation, making the synthesized speech more natural and expressive.
[0067] Speech synthesis is an artificial intelligence technology that converts text information into speech. It can enable computers to "speak" by simulating or synthesizing human voices. For example, in the financial field, by converting product promotion text in financial documents into speech, users who have difficulty reading text can obtain product promotion information through hearing.
[0068] At present, the common speech synthesis method is to use a trained speech synthesis model to convert text information into speech to obtain synthesized speech. However, the synthesized speech generated by this method often only has the characteristics of the speech samples used in model training, which limits the speech characteristics of the synthesized speech. Therefore, how to improve the diversity of synthesized speech features has become a technical problem that needs to be solved urgently.
[0069] Based on this, the embodiments of the present application provide a speech synthesis method and device, an electronic device and a storage medium, aiming to improve the diversity of synthesized speech features.
[0070] The speech synthesis method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.
[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0073] The speech synthesis method provided in the embodiment of the present application relates to the field of speech synthesis technology and is applicable to the field of financial technology. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.
[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0075] Figure 1 is an optional flowchart of the speech synthesis method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.
[0076] Step S101, acquiring speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data;
[0077] Step S102, obtaining a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model;
[0078] Step S103, acquiring speech sample data, and training the initial speech feature extraction sub-model based on the speech sample data to obtain a target speech feature extraction sub-model;
[0079] Step S104, based on the target speech feature extraction sub-model, feature extraction is performed on the target audio data to obtain a target audio feature vector;
[0080] Step S105, based on the text encoding sub-model, text encoding is performed on the speech synthesis text to obtain a text encoding vector;
[0081] Step S106, performing vector mapping on the text coding vector based on the target audio feature vector to obtain a speech coding vector;
[0082] Step S107, generating a sub-model based on the audio signal, and performing speech synthesis on the speech coding vector.
[0083] Steps S101 to S107 shown in the embodiment of the present application obtain speech synthesis data by acquiring speech synthesis text and target audio data, further, obtaining a speech synthesis model including an initial speech feature extraction submodel, a text encoding submodel and an audio signal generation submodel, obtaining speech sample data, and training the initial speech feature extraction submodel based on the speech sample data to obtain a target speech feature extraction submodel, then, according to the target speech feature extraction submodel, feature extraction is performed on the target audio data to obtain a target audio feature vector, and according to the text encoding submodel, text encoding is performed on the speech synthesis text to obtain a text encoding vector, finally, according to the target audio feature vector, vector mapping is performed on the text encoding vector to obtain a speech encoding vector, and then speech synthesis is performed on the speech encoding vector according to the audio signal generation submodel. Therefore, the present application obtains speech synthesis data including speech synthesis text and target audio data, obtains a speech synthesis model including an initial speech feature extraction submodel, a text encoding submodel and an audio signal generation submodel, obtains speech sample data, and trains the initial speech feature extraction submodel based on the speech sample data to obtain a target speech feature extraction submodel, so as to facilitate the subsequent efficient acquisition of the target audio feature vector and ensure the naturalness and diversity of the generated speech. Secondly, according to the target speech feature extraction submodel, feature extraction is performed on the target audio data to obtain the target audio feature vector. According to the text encoding submodel, text encoding is performed on the speech synthesis text to obtain a text encoding vector. According to the target audio feature vector, vector mapping is performed on the text encoding vector to obtain a speech encoding vector, which improves the accuracy of synthesis and ensures that the generated speech can reflect rich emotions and intonation. Finally, according to the audio signal generation submodel, speech synthesis is performed on the speech encoding vector, so as to realize zero-sample generation of new feature speech data and enhance the diversity of speech synthesis.
[0084] In step S101 of some embodiments, the speech synthesis text refers to text data used for speech synthesis, for example, in the financial field, promotional texts of financial products such as insurance, funds, bonds, etc. The target audio data refers to audio data that provides speech features for the synthesized speech.
[0085] The present application can obtain target audio data by recording the audio of a speaker with the required speech features, and then perform text extraction on the text data that needs to be speech synthesized to obtain speech synthesized text. Furthermore, the above-mentioned target audio data and speech synthesized text are merged to obtain speech synthesized data.
[0086] In step S102 of some embodiments, the speech synthesis model is a model that converts text data into audio data. The initial speech feature extraction submodel includes a speaker encoder, a perceptron conditioner and a VQ-VAE, wherein the speaker encoder is used to encode the audio data to realize the vectorization processing of the audio data, the perceptron conditioner is used to extract the speaker's speech features from the encoded audio data, and the VQ-VAE is used to convert the vector representation of the speech features into an audio code. The text encoding submodel includes a GPT model and an LM head, wherein the GPT model is used to encode the speech synthesis text, thereby generating a text encoding representation and a speech encoding representation of the speech synthesis text, and the LM head is used to optimize the text encoding representation of the speech synthesis text to ensure the linguistic fluency and naturalness of the generated speech. The audio signal generation submodel includes a decoder and a discriminator, wherein the decoder is used to convert the speech encoding identifier of the generated speech synthesis text into a continuous audio signal, and the discriminator is used to evaluate the quality of the synthesized speech.
[0087] This application first constructs modules such as speaker encoder, perceptron conditioner, VQ-VAE, GPT model, LM head, decoder and discriminator, then adjusts the parameters of the above modules and assembles the above modules to obtain a speech synthesis model.
[0088] In step S103 of some embodiments, the speech sample data refers to audio data used to train the above-mentioned initial speech feature extraction sub-model.
[0089] The present application can obtain speech sample data for training the initial speech feature extraction submodel by searching the network or querying the database, and after obtaining sufficient speech sample data, adjust the model parameters in the initial speech feature extraction submodel according to the sample data to obtain the target speech feature extraction submodel, so that the target speech feature extraction submodel can be used to accurately extract the speech features of the audio data.
[0090] In step S104 of some embodiments, after obtaining the trained target speech feature extraction sub-model, the target speech feature extraction sub-model can be used to extract the speech features of the target audio data, thereby obtaining a target audio feature vector.
[0091] For details, see Figure 2 In some embodiments, step S104 may include but is not limited to steps S201 to S202:
[0092] Step S201, encoding the target audio data based on the target speech feature extraction sub-model to obtain a target audio encoding vector;
[0093] Step S202: Based on the target speech feature extraction sub-model, the target audio coding vector is quantized to obtain a target audio feature vector.
[0094] In step S201 of some embodiments, the target audio data may be encoded using a trained speaker encoder in the target speech feature extraction submodel to obtain a target audio feature representation, and then the target audio feature representation may be feature optimized using a trained perceptron conditioner in the target speech feature extraction submodel to obtain a target audio encoding vector.
[0095] For details, see Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S303:
[0096] Step S301, based on the target speech feature extraction sub-model, feature recognition is performed on the target audio data to obtain a target audio feature representation;
[0097] Step S302, capturing dynamic features of the target audio feature representation to obtain target audio dynamic features;
[0098] Step S303: Based on the target audio dynamic features, feature optimization is performed on the target audio feature vector representation to obtain a target audio coding vector.
[0099] In step S301 of some embodiments, the target audio data is encoded by a speaker encoder trained in a target speech feature extraction submodel, thereby extracting audio features from the target audio data and obtaining a target audio feature representation.
[0100] In step S302 of some embodiments, the above-mentioned target audio feature representation is further analyzed using the trained perceptron conditioner in the target speech feature extraction submodel to capture the dynamically changing features in the target audio data, for example, the rhythm, intonation, emotion and other features of the target audio data that change over time, thereby generating the target audio dynamic features.
[0101] In step S303 of some embodiments, after obtaining the target audio dynamic features of the target audio data, the target audio feature vector can be adjusted and enhanced using the trained perceptron conditioner in the target speech feature extraction submodel, thereby achieving feature optimization of the target audio feature vector and obtaining the target audio coding vector.
[0102] In steps S301 to S303 shown in the present embodiment, feature recognition is performed on the target audio data according to the target speech feature extraction sub-model to obtain a target audio feature representation. Next, dynamic feature capture is performed on the target audio feature representation to obtain target audio dynamic features. Finally, based on the target audio dynamic features, feature optimization is performed on the target audio feature vector representation to obtain a target audio coding vector. This improves the accuracy of speech feature extraction, thereby ensuring the naturalness of speech features in synthesized speech and helping to improve the naturalness of synthesized speech.
[0103] In step S202 of some embodiments, the trained VQ-VAE in the target speech feature extraction submodel can be used to map the target audio coding vector to a pre-constructed discrete codeword space to obtain discrete features, and then all discrete features are merged to obtain the target audio feature vector.
[0104] For details, see Figure 4 In some embodiments, step S202 may include but is not limited to steps S401 to S403:
[0105] Step S401, based on the target speech feature extraction sub-model, discrete mapping is performed on the target audio coding vector to obtain a discrete codeword, wherein the discrete codeword is a discrete representation of the target audio coding vector;
[0106] Step S402: Based on the discrete codeword, a preset codeword feature mapping table is searched for features to obtain discrete features, wherein the codeword feature mapping table is used to represent a matching relationship between the discrete codeword and the discrete feature;
[0107] Step S403: merge the discrete features to obtain a target audio feature vector.
[0108] In step S401 of some embodiments, the target audio coding vector may be mapped from a continuous feature space to a discrete codeword space using the VQ-VAE trained in the target speech feature extraction submodel, thereby converting the target audio coding vector into a discrete codeword.
[0109] In step S402 of some embodiments, the codeword feature mapping table stores the matching relationship between discrete codewords and discrete features. By querying the codeword feature mapping table, discrete features matching the discrete codewords can be quickly obtained, thereby improving the retrieval efficiency and accuracy of discrete features.
[0110] In step S403 of some embodiments, different discrete features corresponding to different discrete codewords are integrated to obtain a vector that comprehensively identifies the speech features of the target audio data, namely, a target audio feature vector.
[0111] In steps S401 to S403 shown in the present embodiment, the target audio coding vector is discretely mapped according to the target speech feature extraction sub-model to obtain discrete codewords. Secondly, based on the discrete codewords, a feature query is performed on a preset codeword feature mapping table to obtain discrete features. Finally, the discrete features are feature merged to obtain the target audio feature vector, thereby improving the comprehensiveness and naturalness of the speech features, thereby making the synthesized speech more realistic.
[0112] In steps S201 to S202 shown in the present embodiment, the target audio data is encoded according to the target speech feature extraction sub-model to obtain a target audio encoding vector, and then the target audio encoding vector is quantized according to the target speech feature extraction sub-model to obtain a target audio feature vector, which retains the speech features in the target audio data and reduces the complexity of the speech features, thereby improving the efficiency of speech synthesis.
[0113] In step S105 of some embodiments, the GPT model in the above-mentioned text encoding sub-model can be used to encode the speech synthesis text to obtain a vector representation of the speech synthesis text, that is, a text encoding vector.
[0114] For details, see Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S504:
[0115] Step S501, performing text standardization processing on the speech synthesis text to obtain a converted text;
[0116] Step S502, performing word segmentation processing on the converted text to obtain speech synthesis phrases;
[0117] Step S503, encoding the speech synthesis phrase based on the text encoding sub-model to obtain a phrase encoding vector;
[0118] Step S504: Based on the text encoding sub-model, the phrase encoding vectors are merged to obtain the text encoding vector.
[0119] In step S501 of some embodiments, by performing text standardization processing such as removing irrelevant symbols, correcting spelling errors, and unifying terminology on the speech synthesis text, high-quality converted text can be obtained, thereby making the synthesized speech more accurate.
[0120] In step S502 of some embodiments, the text string in the converted text may be decomposed to obtain separate words or phrases, namely, speech synthesis phrases.
[0121] In step S503 of some embodiments, the GPT model in the text encoding sub-model can be used to vectorize the speech synthesis phrases to obtain a vector representation of each speech synthesis phrase, namely, a phrase encoding vector.
[0122] In step S504 of some embodiments, the phrase encoding vectors can be merged according to the input order of the character strings in the converted text using the GPT model in the text encoding sub-model to obtain a vector representation of the speech synthesis text, i.e., a text encoding vector.
[0123] In steps S501 to S504 shown in the present embodiment, text normalization is performed on the speech synthesis text to obtain a converted text, and the converted text is segmented to obtain a speech synthesis phrase. Furthermore, the speech synthesis phrase is encoded according to a text encoding sub-model to obtain a phrase encoding vector, and then the phrase encoding vector is vector merged according to the text encoding sub-model to obtain a text encoding vector, thereby improving the accuracy of the text encoding vector and making the synthesized speech data more accurate.
[0124] In step S106 of some embodiments, a text feature concatenation vector containing speech features is obtained by concatenating the target audio feature vector and the text encoding vector. Further, the text feature concatenation vector is mapped to obtain a speech encoding vector.
[0125] For details, see Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S602:
[0126] Step S601, concatenating the target audio feature vector with the text encoding vector to obtain a text feature concatenated vector;
[0127] Step S602, performing nonlinear transformation on the text feature concatenation vector to obtain a speech coding vector.
[0128] In step S601 of some embodiments, the target audio feature vector and the text encoding vector are dimensionally aligned, and then the dimensionally aligned target audio feature vector and the text encoding vector are concatenated into a longer vector, thereby obtaining a text feature concatenated vector.
[0129] In step S602 of some embodiments, by inputting the text feature splicing vector into a pre-set nonlinear mapping function for calculation, a nonlinear transformation of the text feature splicing vector can be achieved, thereby mapping the text feature splicing vector from the text vector space to the speech vector space to obtain a speech coding vector.
[0130] In steps S601 to S602 shown in the present embodiment, the target audio feature vector and the text encoding vector are vector-concatenated to obtain a text feature concatenated vector. Furthermore, the text feature concatenated vector is nonlinearly transformed to obtain a speech encoding vector, so that the synthesized speech data can have the speech features in the target audio data, thereby improving the diversity of speech synthesis.
[0131] In step S107 of some embodiments, the decoder in the audio signal generation sub-model can be used to perform signal conversion on the speech coding vector to obtain digital audio, and then the digital audio is packaged into a voice data format to obtain synthesized voice data.
[0132] For details, see Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S704:
[0133] Step S701, generating a sub-model based on an audio signal, extracting features from a speech coding vector, and obtaining acoustic features;
[0134] Step S702, based on the acoustic features, generating sound waves for the speech coding vector to obtain target audio sound waves;
[0135] Step S703, converting the target audio sound wave into a digital audio;
[0136] Step S704, encapsulate the digital audio data to obtain synthesized speech data.
[0137] In step S701 of some embodiments, the decoder in the audio signal generation submodel can be used to extract speech features such as pitch, duration, intensity, and formant frequency in the above-mentioned speech coding vector, thereby obtaining the acoustic features in the speech coding vector.
[0138] In step S702 of some embodiments, the speech coding vector can be converted into a signal that can simulate the sound produced by the human vocal tract, that is, the target audio sound wave, based on the acoustic features extracted as described above. It should be noted that the present application can convert the speech coding vector into the target audio sound wave through a vocoder module.
[0139] In step S703 of some embodiments, the target audio sound wave can be converted into digital audio by performing sampling, quantization, encoding and other processing on the target audio sound wave.
[0140] In step S704 of some embodiments, data structures such as file header information and metadata are added to the digital audio to achieve data encapsulation of the digital audio and obtain synthesized speech data, so that the synthesized speech data can be used in various applications, such as voice assistants, audio books, etc.
[0141] In steps S701 to S704 shown in the present embodiment, a sub-model is generated according to a frequency signal, and features are extracted from a speech coding vector to obtain acoustic features. Then, sound waves are generated for the speech coding vector based on the acoustic features to obtain target audio sound waves. Furthermore, the target audio sound waves are format-converted to obtain digital audio, and then the digital audio is data-encapsulated to obtain synthesized speech data. The speech synthesis text can be converted into synthesized speech data having the speech features of the target audio data, thereby realizing a zero-sample speech synthesis process and improving the diversity of speech synthesis.
[0142] The present application obtains speech synthesis data including speech synthesis text and target audio data, obtains a speech synthesis model including an initial speech feature extraction submodel, a text encoding submodel and an audio signal generation submodel, obtains speech sample data, and trains the initial speech feature extraction submodel based on the speech sample data to obtain a target speech feature extraction submodel, so as to facilitate the subsequent efficient acquisition of the target audio feature vector and ensure the naturalness and diversity of the generated speech. Secondly, according to the target speech feature extraction submodel, feature extraction is performed on the target audio data to obtain the target audio feature vector. According to the text encoding submodel, text encoding is performed on the speech synthesis text to obtain the text encoding vector. According to the target audio feature vector, vector mapping is performed on the text encoding vector to obtain the speech encoding vector, so as to improve the accuracy of synthesis and ensure that the generated speech can reflect rich emotions and intonation. Finally, according to the audio signal generation submodel, speech synthesis is performed on the speech encoding vector, so as to realize the zero-sample generation of new feature speech data and enhance the diversity of speech synthesis.
[0143] See also Figure 8 The present application also provides a speech synthesis device, which can implement the above-mentioned speech synthesis method, and the device includes:
[0144] The data acquisition module 801 is used to acquire speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data;
[0145] A model acquisition module 802 is used to acquire a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model;
[0146] The model training module 803 is used to obtain speech sample data and train the initial speech feature extraction sub-model based on the speech sample data to obtain the target speech feature extraction sub-model;
[0147] A feature extraction module 804 is used to extract features from target audio data based on a target speech feature extraction sub-model to obtain a target audio feature vector;
[0148] A text encoding module 805 is used to perform text encoding on the speech synthesis text based on the text encoding sub-model to obtain a text encoding vector;
[0149] A speech coding module 806 is used to perform vector mapping on the text coding vector based on the target audio feature vector to obtain a speech coding vector;
[0150] The speech synthesis module 807 is used to generate a sub-model based on the audio signal, perform speech synthesis on the speech coding vector, and obtain synthesized speech data.
[0151] The specific implementation of the speech synthesis device is basically the same as the specific embodiment of the above-mentioned speech synthesis method, and will not be repeated here.
[0152] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speech synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0153] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0154] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0155] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech synthesis method of the embodiment of this application;
[0156] Input / output interface 903, used to implement information input and output;
[0157] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0158] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0159] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0160] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech synthesis method is implemented.
[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0162] The speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of the present application obtain speech synthesis data by acquiring speech synthesis text and target audio data, and further obtain a speech synthesis model including an initial speech feature extraction submodel, a text encoding submodel and an audio signal generation submodel, obtain speech sample data, and train the initial speech feature extraction submodel based on the speech sample data to obtain a target speech feature extraction submodel, then perform feature extraction on the target audio data according to the target speech feature extraction submodel to obtain a target audio feature vector, and perform text encoding on the speech synthesis text according to the text encoding submodel to obtain a text encoding vector, and finally, perform vector mapping on the text encoding vector according to the target audio feature vector to obtain a speech encoding vector, and then perform speech synthesis on the speech encoding vector according to the audio signal generation submodel.
[0163] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0164] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0165] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0166] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0167] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0168] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0169] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0170] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0173] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.
Claims
1. A speech synthesis method, characterized in that: The method comprises: Acquire speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data; Acquire a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model; Acquire speech sample data, and train the initial speech feature extraction sub-model based on the speech sample data to obtain a target speech feature extraction sub-model; Based on the target speech feature extraction sub-model, feature extraction is performed on the target audio data to obtain a target audio feature vector; Based on the text encoding sub-model, text encoding is performed on the speech synthesis text to obtain a text encoding vector; Based on the target audio feature vector, performing vector mapping on the text encoding vector to obtain a speech encoding vector; A sub-model is generated based on the audio signal to perform speech synthesis on the speech coding vector.
2. The method according to claim 1, characterized in that The generating a sub-model based on the audio signal and performing speech synthesis on the speech coding vector to obtain synthesized speech data includes: Based on the audio signal generating sub-model, feature extraction is performed on the speech coding vector to obtain acoustic features; Based on the acoustic features, generating sound waves for the speech coding vector to obtain target audio sound waves; Converting the target audio sound wave into a digital audio format; The digital audio is data packaged to obtain the synthesized speech data.
3. The method according to claim 1, characterized in that The step of extracting features from the target audio data based on the target speech feature extraction sub-model to obtain a target audio feature vector includes: Based on the target speech feature extraction sub-model, encoding the target audio data to obtain a target audio encoding vector; Based on the target speech feature extraction sub-model, the target audio coding vector is quantized to obtain the target audio feature vector.
4. The method according to claim 3, characterized in that The step of encoding the target audio data based on the target speech feature extraction sub-model to obtain a target audio encoding vector comprises: Based on the target speech feature extraction sub-model, feature recognition is performed on the target audio data to obtain a target audio feature representation; Capturing dynamic features of the target audio feature representation to obtain target audio dynamic features; Based on the target audio dynamic features, feature optimization is performed on the target audio feature vector representation to obtain the target audio encoding vector.
5. The method according to claim 3, characterized in that: The step of performing quantization processing on the target audio coding vector based on the target speech feature extraction sub-model to obtain the target audio feature vector comprises: Based on the target speech feature extraction sub-model, the target audio coding vector is discretely mapped to obtain a discrete codeword, wherein the discrete codeword is a discrete representation of the target audio coding vector; Based on the discrete codeword, a feature query is performed on a preset codeword feature mapping table to obtain a discrete feature, wherein the codeword feature mapping table is used to represent a matching relationship between the discrete codeword and the discrete feature; The discrete features are combined to obtain the target audio feature vector.
6. The method according to any one of claims 1 to 5, characterized in that: The step of performing vector mapping on the text encoding vector based on the target audio feature vector to obtain a speech encoding vector includes: Performing vector concatenation of the target audio feature vector and the text encoding vector to obtain a text feature concatenation vector; The text feature concatenation vector is subjected to nonlinear transformation to obtain the speech coding vector.
7. The method according to any one of claims 1 to 5, characterized in that: The step of performing text encoding on the speech synthesis text based on the text encoding sub-model to obtain a text encoding vector includes: Performing text standardization processing on the speech synthesis text to obtain a converted text; Performing word segmentation processing on the converted text to obtain speech synthesis phrases; Based on the text encoding sub-model, encoding the speech synthesis phrase to obtain a phrase encoding vector; Based on the text encoding sub-model, the phrase encoding vectors are merged to obtain the text encoding vector.
8. A speech synthesis device, characterized in that: The device comprises: A data acquisition module, used to acquire speech synthesis data, wherein the speech synthesis data includes speech synthesis text and target audio data; A model acquisition module, used to acquire a speech synthesis model, wherein the speech synthesis model includes an initial speech feature extraction sub-model, a text encoding sub-model and an audio signal generation sub-model; A model training module, used for acquiring speech sample data, and training the initial speech feature extraction sub-model based on the speech sample data to obtain a target speech feature extraction sub-model; A feature extraction module, used to extract features from the target audio data based on the target speech feature extraction sub-model to obtain a target audio feature vector; A text encoding module, used for performing text encoding on the speech synthesis text based on the text encoding sub-model to obtain a text encoding vector; A speech coding module, used for performing vector mapping on the text coding vector based on the target audio feature vector to obtain a speech coding vector; The speech synthesis module is used to generate a sub-model based on the audio signal, perform speech synthesis on the speech coding vector, and obtain synthesized speech data.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.