Model training method and device, speech synthesis method and device, and storage medium

Abstract

Embodiments of the present application provide a model training method and device, a speech synthesis method, equipment and a storage medium, and belong to the technical field of artificial intelligence. The method comprises: obtaining sample speech feature data and sample phoneme data; inputting the sample speech feature data into a preset original feature generation model to perform feature expansion to obtain preliminary speech feature data; performing loss calculation on the sample speech feature data and the preliminary speech feature data to obtain a feature loss value; performing parameter adjustment on the original feature generation model according to the feature loss value to obtain a target feature generation model; inputting the sample speech feature data into the target feature generation model to perform feature expansion to obtain target speech feature data; updating original sample data according to the target speech feature data to obtain target sample data; and training a preset original speech synthesis model according to the target sample data to obtain a target speech synthesis model. The embodiments of the present application can obtain a speech synthesis model with better synthesis effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a model training method and device, a speech synthesis method, equipment and a storage medium. BACKGROUND

[0002] A machine learning model needs a large amount of data for training when learning. For example, a large amount of speech resources need to be collected as sample data for model training in the field of speech synthesis, but speech resources involving multiple language types are less, and how to train a speech synthesis model with better synthesis effect through a small amount of speech samples of different language types is a problem to be solved at present. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a model training method and device, a speech synthesis method, equipment and a storage medium, aiming to train a speech synthesis model with better speech synthesis effect in the scene of a small amount of speech samples.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a model training method, which comprises:

[0005] Obtaining original sample data; wherein the original sample data comprises sample speech feature data and sample phoneme data;

[0006] Inputting the sample speech feature data into a preset original feature generation model for feature expansion processing to obtain preliminary speech feature data;

[0007] Calculating the loss of the sample speech feature data and the preliminary speech feature data to obtain a feature loss value;

[0008] Adjusting the parameters of the original feature generation model according to the feature loss value until the feature loss value is minimum to obtain a target feature generation model;

[0009] Inputting the sample speech feature data into the target feature generation model for feature expansion processing to obtain target speech feature data;

[0010] Constructing target sample data according to the sample speech feature data, the target speech feature data and the sample phoneme data;

[0011] Training a preset original speech synthesis model according to the target sample data to obtain a target speech synthesis model.

[0012] In some embodiments, the original feature generation model comprises: a first encoder, a first decoder and a network layer; the feature augmentation processing of the sample speech feature data input into the preset original feature generation model comprises:

[0013] The sample speech feature data is input into the first encoder for encoding processing to obtain a preliminary feature vector;

[0014] The preliminary feature vector is converted by the network layer according to a preset codebook to obtain an embedded candidate feature vector;

[0015] The preset speaker object information is embedded into the embedded candidate feature vector by the network layer to obtain a decoding candidate feature vector;

[0016] The decoding candidate feature vector is decoded by the first decoder to obtain the preliminary speech feature data.

[0017] In some embodiments, the conversion processing of the preliminary feature vector by the network layer according to a preset codebook to obtain an embedded candidate feature vector comprises:

[0018] The discrete variable of the preset codebook is obtained by the network layer to obtain a codebook vector;

[0019] The preliminary feature vector is mapped by the network layer according to the codebook vector to obtain the embedded candidate feature vector.

[0020] In some embodiments, the target sample data is constructed according to the sample speech feature data, the target speech feature data and the sample phoneme data, comprising:

[0021] The sample speech feature data and the target speech feature data are merged to obtain updated speech feature data;

[0022] The updated speech feature data and the sample phoneme data are filtered according to a preset data mapping relationship to obtain audio pairing data;

[0023] The target sample data is constructed according to the audio pairing data, the updated speech feature data and the sample phoneme data.

[0024] In some embodiments, the model training of the preset original speech synthesis model according to the target sample data to obtain a target speech synthesis model comprises:

[0025] The sample phoneme data is input into the original speech synthesis model for speech synthesis to obtain speech synthesis prediction data;

[0026] The update voice feature data is filtered according to the audio pairing data and the sample phoneme data, and voice feature verification data is obtained.

[0027] The original voice synthesis model is adjusted in parameters according to the voice feature verification data and the voice synthesis prediction data, and the target voice synthesis model is obtained.

[0028] In some embodiments, the original voice synthesis model comprises a second encoder, a second decoder and a vocoder; and the inputting of the sample phoneme data into the original voice synthesis model for voice synthesis to obtain voice synthesis prediction data comprises:

[0029] The sample phoneme data is input into the second encoder for encoding processing to obtain a preliminary voice vector;

[0030] The preliminary voice vector is subjected to attention processing according to preset speaker object information to obtain a decoding candidate voice vector;

[0031] The decoding candidate voice vector is subjected to decoding processing by the second decoder to obtain voice spectrum data;

[0032] The voice spectrum data is subjected to waveform calculation by the vocoder to obtain the voice synthesis prediction data.

[0033] To achieve the above object, a second aspect of the embodiment of the present application proposes a voice synthesis method, which comprises:

[0034] Obtaining target phoneme data;

[0035] Inputting the target phoneme data into a target voice synthesis model; wherein the target voice synthesis model is obtained by the model training method as described in the first aspect;

[0036] Performing voice synthesis on the target phoneme data by the target voice synthesis model to obtain target voice synthesis data; wherein the target voice synthesis data is waveform data based on mel spectrum.

[0037] To achieve the above object, a third aspect of the embodiment of the present application proposes a model training device, which comprises:

[0038] A sample obtaining module is configured to obtain original sample data; wherein the original sample data comprises sample voice feature data and sample phoneme data;

[0039] A data input module is configured to input the sample voice feature data into a preset original feature generation model for feature expansion processing to obtain preliminary voice feature data;

[0040] a loss calculation module, configured to perform loss calculation on the sample voice feature data and the preliminary voice feature data to obtain a feature loss value;

[0041] a parameter adjustment module, configured to perform parameter adjustment on the original feature generation model according to the feature loss value until the feature loss value is minimum, to obtain a target feature generation model;

[0042] a data expansion module, configured to input the sample voice feature data into the target feature generation model to perform feature expansion processing, to obtain target voice feature data;

[0043] a data collection module, configured to construct target sample data according to the sample voice feature data, the target voice feature data and the sample phoneme data;

[0044] a model training module, configured to perform model training on a preset original voice synthesis model according to the target sample data, to obtain a target voice synthesis model.

[0045] To achieve the above object, a fourth aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, a computer program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the computer program is executed by the processor to realize the method of the first aspect or the method of the second aspect.

[0046] To achieve the above object, a fifth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method of the first aspect or the method of the second aspect.

[0047] The model training method and device, the voice synthesis method, the equipment and the storage medium provided by the present application train the original feature generation model according to the sample voice feature data to obtain the target feature generation model, and the target feature generation model performs feature expansion on the sample voice feature data to obtain the target voice feature data, so as to expand the voice feature data in the case of a small amount of sample data. Then, the target sample data is constructed according to the target voice feature data, the sample voice feature data and the sample phoneme data. After expanding the sample data, the target voice synthesis model is obtained by performing model training on the original voice synthesis model according to the target sample data, so as to construct the voice synthesis model with better voice synthesis effect in the case of a small amount of sample data. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the model training method provided by the embodiment of the present application;

[0049] Figure 2 is a flowchart of step S102 in Figure 1

[0050] Figure 3 is a flowchart of step S202 in Figure 2

[0051] Figure 4 is a flowchart of step S106 in Figure 1

[0052] Figure 5 is a flowchart of step S107 in Figure 1

[0053] Figure 6 is a flowchart of step S502 in Figure 5

[0054] Figure 7 is a flowchart of a speech synthesis method provided by an embodiment of the present application;

[0055] Figure 8 is a structural schematic diagram of a model training apparatus provided by an embodiment of the present application;

[0056] Figure 9 is a structural schematic diagram of a speech synthesis apparatus provided by an embodiment of the present application;

[0057] Figure 10 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0059] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0061] ​​​​​First, the meanings of several terms involved in the present application are analyzed:

[0062] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.), and NLP is a branch of artificial intelligence, which is an interdisciplinary subject of computer science and linguistics, and is also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0064] Few-shot Learning: Few-shot Learning is the application of Meta Learning in the field of supervised learning. Meta Learning, also known as learning to learn, is an algorithm designed to enable models to learn "learning" and handle similar types of tasks, rather than just a single classification task. In Meta Learning, the data set is divided into different meta tasks in the meta training phase to learn the model's generalization ability under class changes. In the meta testing phase, facing new classes, the existing model can be completed without changing the classification. In Few-shot Learning, there is a term called N-way K-shot problem. Formally, the few-shot training set contains many classes, and each class has multiple samples. In the training phase, N classes are randomly selected from the training set, each with K samples (N*K data in total), to construct a meta-task as the support set input of the model; then a batch of samples is selected from the remaining data in the N classes as the prediction object of the model (batch set). The model is required to learn how to distinguish between the N classes from the N*K data. Such a task is called an N-way K-shot problem.

[0065] Text to Speech (TTS): Text to Speech is a technology that converts text into speech, giving machines the ability to speak as naturally as humans. It is an important part of human-computer voice interaction. Speech synthesis systems are widely used in various scenarios in life, including voice dialogue systems; intelligent voice assistants such as Siri and Xunfei Yudian; telephone information query systems; car navigation, voice e-books and other auxiliary applications; language learning; real-time information broadcasting systems in airports and stations; information acquisition and communication for the visually or hearing impaired, etc. Speech synthesis is a technology that produces artificial speech through mechanical and electronic methods. TTS technology (also known as text-to-speech technology) belongs to speech synthesis, which is a technology that converts computer-generated or externally input text information into understandable and fluent Chinese spoken output.

[0066] VQ-VAE: VQ-VAE is a generative model of discrete latent variables. VQ-VAE has two important differences compared to VAE: first, VQ-VAE adopts discrete latent variables, rather than continuous latent variables as VAE; then VQ-VAE needs to train a separate autoregressive-based model such as PixelCNN to learn the prior, rather than using a fixed prior (standard normal distribution) as VAE. In addition, VQ-VAE is also a powerful unsupervised representation learning model, and the discrete code it learns has strong representation ability. A VAE model includes three parts: posterior distribution, prior distribution, and likelihood. Among them, the posterior distribution is learned by an encoder network, the likelihood is learned by a decoder network, and the prior distribution uses a standard normal distribution with fixed parameters.

[0067] Mel-spectrogram: In speech analysis, synthesis, and conversion, the first step is often to extract speech feature parameters. Mel-spectrogram is commonly used in speech tasks using machine learning methods. We usually listen to music and speech in the time domain. Using Fourier transform on the time domain signal can get the frequency domain representation of the signal. Using short-time Fourier transform to get the frequency spectrum, and then taking the modulus square can get the following power spectrum. Based on the frequency spectrum, we can get the mel-spectrogram. Multiply the power spectrum by the mel filter bank to get the mel-spectrogram.

[0068] With the development of intelligent technology, speech synthesis technology is widely used in various intelligent fields. For example, in the field of intelligent tourism, the intelligent robot can accurately broadcast navigation content and scenic spot information through the voice broadcast function, and can accurately answer user inquiries. Therefore, speech synthesis technology has become one of the important contents of human-computer voice interaction. Among them, speech synthesis generates speech synthesis data through a speech synthesis model, so it is necessary to construct a speech synthesis model before speech synthesis. Generally, speech sample data is obtained to train the speech synthesis model. However, in related technologies, there are more Mandarin or English resources. If you need to construct a dialect type speech synthesis, you need to obtain a large amount of dialect resources. However, obtaining dialect resources requires a lot of manpower and time, so how to construct a speech synthesis model with better speech synthesis effect through a small amount of speech sample data is a problem to be solved at present.

[0069] Therefore, an embodiment of the present application provides a model training method and device, a speech synthesis method, equipment and a storage medium, which aims to construct an accurate speech synthesis model through a small amount of speech data resources.

[0070] The model training method and device, the speech synthesis method, the equipment and the storage medium provided by the embodiments of the present application are described as follows. First, the model training method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0072] The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, operation / interaction systems, mechatronics, etc. The software technologies of artificial intelligence mainly include computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0073] The model training method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The model training method provided by the embodiments of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, etc. The software can be an application for implementing the model training method, but is not limited to the above forms.

[0074] The application is operable in a multitude of generic or specific computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0075] It should be noted that in each of the specific embodiments of the present application, when it is necessary to perform relevant processing according to user information, user behavior data, user history data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards in relevant countries and regions. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0076] Figure 1 is an optional flowchart of the model training method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to including steps S101 to S107.

[0077] Step S101, obtaining original sample data; wherein the original sample data includes sample speech feature data and sample phoneme data;

[0078] Step S102, inputting the sample speech feature data into a preset original feature generation model for feature expansion processing to obtain preliminary speech feature data;

[0079] Step S103, performing loss calculation on the sample speech feature data and the preliminary speech feature data to obtain a feature loss value;

[0080] Step S104, adjusting parameters of the original feature generation model according to the feature loss value until the feature loss value is minimum to obtain a target feature generation model;

[0081] Step S105, input the sample voice feature data to the target feature generation model for feature expansion processing to obtain target voice feature data.

[0082] Step S106, construct target sample data according to the sample voice feature data, the target voice feature data and the sample phoneme data.

[0083] Step S107, model training is performed on the preset original voice synthesis model according to the target sample data to obtain a target voice synthesis model.

[0084] The steps S101 to S107 shown in the embodiments of the present application, by obtaining sample voice feature data and sample phoneme data, inputting the sample voice feature data to the preset original feature generation model for feature expansion processing to obtain preliminary voice feature data, and performing loss calculation on the sample voice feature data and the preliminary voice feature data to obtain a feature loss value, adjusting the parameters of the original feature generation model according to the feature loss value until the feature loss value is minimum to obtain a target feature generation model. Then input the sample voice feature data to the target feature generation model for feature expansion processing to obtain target voice feature data, then construct target sample data from the sample voice feature data, the target voice feature data and the sample phoneme data, and perform model training on the preset original voice synthesis model according to the target sample data to obtain a target voice synthesis model. Therefore, by constructing a target feature generation model, and performing feature expansion processing on the sample voice feature data by the target feature generation model to obtain target voice feature data, the voice feature data is expanded in the case of a small amount of sample data, that is, a voice synthesis model with better voice synthesis effect is constructed in the scene of few sample learning.

[0085] In step S101 of some embodiments, original sample data is obtained, and the original sample data is obtained through a sample database. The original sample data can also be obtained by other means, which is not limited. The original sample data includes sample phoneme data and sample voice feature data, and the language types of the sample voice feature data include but are not limited to Mandarin, English, Cantonese, Shanghai dialect, Hakka, and the sample voice feature data is classified and stored in the corresponding database according to the language type, so as to extract the sample voice feature data from the corresponding database according to the language requirement of voice synthesis. Wherein, the sample phoneme data is obtained by obtaining sample text data and performing semantic understanding on the sample text data.

[0086] Please refer to Figure 2In some embodiments, the original feature generation model comprises a first encoder, a first decoder and a network layer; the original feature generation model is a VQ-VAE model, and the VQ-VAE model serves as an extractor of the discretized speech unit, and the VQ-VAE model comprises a first encoder, a first decoder and a network layer, the first encoder is capable of encoding sample speech feature data into a latent variable space, and the first decoder is capable of feature reconstruction to obtain new sample speech feature data, thereby expanding the sample speech feature data.

[0087] In some embodiments, step S102 can include, but is not limited to, steps S201 to S204:

[0088] In step S201, the sample speech feature data is input into the first encoder for encoding processing to obtain a preliminary feature vector.

[0089] In step S202, the preliminary feature vector is converted by the network layer using a preset codebook to obtain an embedded candidate feature vector.

[0090] In step S203, the preset speaker object information is embedded into the embedded candidate feature vector by the network layer to obtain a decoding candidate feature vector.

[0091] In step S204, the decoding candidate feature vector is decoded by the first decoder to obtain preliminary speech feature data.

[0092] The steps S201 to S204 shown in the embodiments of the present application input the sample speech feature data into the first encoder for encoding processing to obtain a preliminary feature vector, and convert the preliminary feature vector by the network layer using a preset codebook to obtain an embedded candidate feature vector, and embed the preset speaker object information into the embedded candidate feature vector to obtain a decoding candidate feature vector, and finally decode the decoding candidate feature vector by the first decoder to obtain preliminary speech feature data. Therefore, the preliminary speech feature data is reconstructed from the sample speech feature data to expand the sample speech feature data, making the expansion of the sample speech feature data simple.

[0093] In step S201 of some embodiments, the sample speech feature data is input into the first encoder, and the first encoder encodes the sample speech feature data into a latent variable space to obtain a preliminary feature vector. For example, if the sample speech feature data is x and the preliminary feature vector is z, the formula for mapping the sample speech feature data to the latent variable space is z e (x) = z, where z e represents the encoding of x into the z space to obtain the preliminary feature vector z, making the operation of encoding the sample speech feature data into the preliminary feature vector simple.

[0094] In step S202 of some embodiments, the sample speech feature data is encoded by the first encoder to obtain a preliminary feature vector, the preliminary feature vector is represented by an intermediate latent variable, and the preliminary feature vector is a continuous variable. If the continuous variable needs to be converted into a discrete variable, a preset codebook needs to be preset, and after the preset codebook is initialized, the preliminary feature vector is represented by a discrete variable in the preset codebook to obtain an embedded candidate feature vector, that is, a vector-quantized embedded candidate feature vector. For example, if the size of the preset codebook is 50, the preset codebook is set as a vector set with a quantity of 50, and then the continuous preliminary feature vector is represented as the vector set with a quantity of 50 to obtain the embedded candidate feature vector.

[0095] Specifically, the preliminary feature vector is replaced by the embedded candidate feature vector according to the preset codebook, that is, In the formula, z i，j is the preliminary feature vector, e k is a code word of the preset codebook. According to the preset codebook, the code word e i，j corresponding to the preliminary feature vector z k is found, and the preliminary feature vector is replaced by the code word one by one, so that the preliminary feature vector z is replaced by e k , so as to convert the continuous variable into a discrete variable, so that the generated preliminary speech feature data is more matched with the sample speech feature data after being decoded by the first decoder.

[0096] In step S203 of some embodiments, in order to make the reconstructed preliminary speech feature data more consistent with the speaking object, the preset speaking object information is embedded into the embedded candidate feature vector by the network layer to obtain a decoding candidate feature vector, so as to reconstruct the decoding candidate feature vector consistent with the target speaking object.

[0097] In step S204 of some embodiments, after the decoding candidate feature vector is output, the network layer inputs the decoding candidate feature vector to the first decoder, and then the first decoder decodes the discretized decoding candidate feature vector, reconstructs the speech feature to obtain the preliminary speech feature data. Therefore, by reconstructing the speech feature, a small amount of speech feature is increased in quantity, and a large amount of speech feature data does not need to be artificially collected, thereby saving the time and effort spent on artificially collecting training samples.

[0098] Please refer to Figure 3 In some embodiments, step S202 can include but is not limited to steps S301 to S302:

[0099] In step S301, the discrete variable of the preset codebook is obtained by the network layer to obtain a codebook vector;

[0100] Step S302, the preliminary feature vector is mapped to the codebook vector by the network layer to obtain an embedded candidate feature vector.

[0101] The steps S301 to S302 shown in the embodiments of the present application obtain the discrete variables of the preset codebook to obtain the codebook vector, and then map the preliminary feature vector to the codebook vector, that is, replace the preliminary feature vector according to the format of the codebook vector to obtain the embedded candidate feature vector. Therefore, by mapping the continuous preliminary feature vector to the discrete codebook vector, a higher level of hidden representation is obtained, that is, a discrete embedded candidate feature vector is obtained.

[0102] In step S301 of some embodiments, after the preliminary feature vector is input into the network layer, the discrete variables of the preset codebook are obtained, that is, the code words of the preset codebook are obtained to obtain the codebook vector. For example, if the preset codebook is e=C*D, C is the number of embeddings, and D is the size of each embedding. The discrete variables of the preset codebook are extracted to obtain the codebook vector C*D, so as to map the preliminary feature vector according to the codebook vector.

[0103] In step S302 of some embodiments, after the codebook vector is obtained, the preliminary feature vector is mapped to the discrete codebook vector to obtain the embedded candidate feature vector, so that a higher level of hidden representation is generated before the preset speaker information is embedded into the embedded candidate feature vector. Therefore, after the embedded candidate feature vector is obtained, the preliminary speech feature data generated by the first decoder is more similar to the sample speech feature data, which not only can expand the sample speech feature data, but also can generate target speech feature data similar to the sample speech feature data.

[0104] In step S103 of some embodiments, the sample speech feature data and the preliminary speech feature data are subjected to loss calculation to obtain a feature loss value. The feature loss value is the reconstruction loss of the speech feature data, in addition to which a codebook loss and an output loss also need to be calculated. The reconstruction loss is the MSE loss between the reconstructed preliminary speech feature data and the sample speech feature data, the codebook loss is used to update the preset codebook to build a more accurate preset codebook, and the output loss is the output loss of the first encoder, so as to adjust the first encoder according to the output loss, so that the preliminary feature vector output by the first encoder is closer to the preset codebook. For example, by calculating the feature loss value, and the calculation formula of the feature loss value is shown in formula (1):

[0105] L = log p (x | Z e (x))-sg(e) (1)

[0106] In the formula, Z e (x) is the sample speech feature data, and sg(e) is the preliminary speech feature data.

[0107] In step S104 of some embodiments, the parameters of the original feature generation model are adjusted according to the feature loss value, and the preliminary speech feature data is obtained by continuing the feature expansion processing of the sample speech feature data through the original feature generation model after the parameter adjustment, and the feature expansion processing, the feature loss value calculation and the parameter adjustment are repeatedly performed until the calculated feature loss value is minimum, and then the parameter adjustment of the original feature generation model is stopped to obtain the target feature generation model. The constructed target feature generation model is an unsupervised speech model, which can expand the sample speech feature data through the unsupervised speech model, so as to construct a speech synthesis model with better speech synthesis effect even in the case of a small amount of sample speech feature data.

[0108] In step S105 of some embodiments, the target speech feature data is obtained by inputting the sample speech feature data into the target feature generation model, and the target speech feature data is close to the sample speech feature data, which not only realizes the expansion of the sample speech feature data, but also does not affect the consistency of the sample speech feature data, so as to construct the speech synthesis model more accurately according to the sample speech feature data and the target speech feature data.

[0109] It should be noted that the target feature generation model is consistent with the specific structure of the sample speech feature data, and the target feature generation model includes a first encoder, a first decoder and a network layer, and the target feature generation model encodes, transforms, embeds speaker object information and decodes the sample speech feature data to obtain the target speech feature data. Since the target feature generation model is a trained feature generation model, the target speech feature data is generated through the target feature generation model, and the target speech feature data is closer to the sample speech feature data, so that the speech feature data is kept approximate under the premise of expanding the sample speech feature data, and the training of the speech synthesis model is also more accurate.

[0110] Please refer to Figure 4 In some embodiments, step S106 can include but is not limited to steps S401 to S403:

[0111] Step S401, merging the sample speech feature data and the target speech feature data to obtain updated speech feature data;

[0112] Step S402, filtering the updated speech feature data and the sample phoneme data according to a preset data mapping relationship to obtain audio pairing data;

[0113] Step S403, constructing target sample data according to audio pairing data, updated speech feature data and sample phoneme data.

[0114] The steps S401 to S403 shown in the embodiments of the present application, the sample speech feature data and the target speech feature data are merged to obtain the updated speech feature data, and the updated speech feature data and the sample phoneme data are filtered according to the preset data mapping relationship to obtain the audio pairing data. The audio pairing data is the pairing relationship of the updated speech feature data and the sample phoneme data. Finally, the audio pairing data, the updated speech feature data and the sample phoneme data are constructed to obtain the target sample data, so that the target sample data is simple to construct, so as to train the original speech synthesis model according to the target sample data. Among them, the updated speech feature data corresponding to the sample phoneme data can be determined through the audio pairing data, so as to determine the input and output of the original speech synthesis model according to the updated speech feature data corresponding to the sample phoneme data.

[0115] In step S401 of some embodiments, since the target speech feature data is the speech feature data similar to the sample speech feature data, the target speech feature data and the sample speech feature data are merged to obtain the updated speech feature data, so as to expand the sample speech feature data. Among them, the language types of the sample speech feature data include any one of the following: Mandarin, English, Cantonese, Shanghai dialect, Hakka, then the sample speech feature data of each language type is processed by the target feature generation model to obtain the target speech feature data, and the sample speech feature data and the target speech feature data of the same language type are merged to obtain the updated speech feature data.

[0116] In step S402 of some embodiments, the preset data mapping relationship records the mapping relationship between the speech feature data and the phoneme data, so the updated speech feature data and the sample phoneme data are filtered according to the data mapping relationship, that is, the updated speech feature data corresponding to each sample phoneme data is determined, and the audio pairing data is determined according to the updated speech feature data corresponding to the sample phoneme data. For example, it is determined that the sample phoneme data is A and B respectively, according to the preset data mapping relationship, it is determined that the updated speech feature data corresponding to A is Y1, and the updated speech feature data corresponding to B is Y2, then the audio pairing data is constructed as <A, Y1>, <B, Y2>, so as to determine the updated speech feature data corresponding to the sample phoneme data according to the audio pairing data, so as to train the original speech synthesis model.

[0117] In step S403 of some embodiments, the target sample data is constructed according to the audio pairing data, the updated speech feature data, and the sample phoneme data, so as to obtain the expanded target sample data, so as to realize the speech synthesis task in the scene of a small amount of sample resources, and to construct a more accurate target speech synthesis model.

[0118] Please refer to Figure 5 In some embodiments, step S107 can include, but is not limited to, steps S501 to S503:

[0119] In step S501, the sample phoneme data is input into the original speech synthesis model for speech synthesis to obtain speech synthesis prediction data.

[0120] In step S502, the updated speech feature data is filtered according to the audio pairing data and the sample phoneme data to obtain speech feature verification data.

[0121] In step S503, the original speech synthesis model is adjusted according to the speech feature verification data and the speech synthesis prediction data to obtain a target speech synthesis model.

[0122] The steps S501 to S503 shown in the embodiments of the present application input the sample phoneme data into the original speech synthesis model, the original speech synthesis model performs speech synthesis according to the sample phoneme data to obtain speech synthesis prediction data, and finds the corresponding updated speech feature data in the audio pairing data according to the sample phoneme data to obtain speech feature verification data. The original speech synthesis model is adjusted according to the speech feature verification data and the speech synthesis prediction data, so that the speech synthesis prediction data approximates the speech feature verification data to obtain a target speech synthesis model. Therefore, through the expanded updated speech feature data, the original speech synthesis model can be trained in the case of a small amount of sample resources to obtain a target speech synthesis model with better speech synthesis effect.

[0123] In step S501 of some embodiments, the sample phoneme data is input into the original speech synthesis model, and the original speech synthesis model performs speech synthesis according to the sample phoneme data to obtain speech synthesis prediction data. The speech synthesis prediction data is waveform data based on mel spectrum, so as to determine the effect of speech synthesis of the original speech synthesis model according to the speech synthesis prediction data.

[0124] It should be noted that the original speech synthesis model is a Seq2seq model, and the Seq2seq model generates speech synthesis prediction data by inputting sample phoneme data. The Seq2seq model of the present application is applied to a question and answer system to generate speech synthesis data according to the phoneme data input by a user, and to realize intelligent question and answer.

[0125] In step S502 of some embodiments, the audio pairing data includes a pairing relationship between the sample phoneme data and the updated speech feature data, and the updated speech feature data is filtered according to the audio pairing data and the sample phoneme data to obtain speech feature verification data corresponding to the sample phoneme data, and whether the speech synthesis prediction data is correct is determined according to the speech feature verification data, so as to adjust the original speech synthesis model according to the speech feature verification data and the speech synthesis prediction data.

[0126] In step S503 of some embodiments, since the speech synthesis prediction data is waveform data based on mel spectrum, the sample speech feature data is input into the vocoder to obtain waveform data based on mel spectrum, and the original speech synthesis model is adjusted according to the matching of the two waveform data to obtain a target speech synthesis model, so that the speech synthesis prediction data output by the target speech synthesis model is closer to the sample speech feature data, thereby improving the effect of speech synthesis.

[0127] Please refer to Figure 6 In some embodiments, the original speech synthesis model includes a second encoder, a second decoder, and a vocoder, and step S502 includes but is not limited to steps S601 to S604:

[0128] In step S601, the sample phoneme data is input into the second encoder for encoding processing to obtain a preliminary speech vector.

[0129] In step S602, the preliminary speech vector is processed according to the preset speaker information to obtain a decoding candidate speech vector.

[0130] In step S603, the decoding candidate speech vector is decoded by the second decoder to obtain speech spectrum data.

[0131] In step S604, the speech spectrum data is calculated by the vocoder to obtain speech synthesis prediction data.

[0132] The steps S601 to S604 shown in the embodiments of the present application input the sample phoneme data into the second encoder to encode the sample phoneme data into a preliminary speech vector, represent the sample phoneme data in the form of a vector, process the preliminary speech vector according to the preset speaker information to improve the weight of the vector corresponding to the speaker information in the preliminary speech vector to obtain a decoding candidate speech vector, decode the decoding candidate speech vector by the second decoder to obtain speech spectrum data, and calculate the speech spectrum data by the vocoder to obtain speech synthesis prediction data, so as to directly generate speech synthesis prediction data from phoneme data, making the speech synthesis operation simple and more consistent with the speaker.

[0133] In step S601 of some embodiments, the sample phoneme data is input into a second encoder, and the second encoder encodes the sample phoneme data to convert the sample phoneme data into a fixed-length vector to obtain a preliminary speech vector.

[0134] In step S602 of some embodiments, after the second encoder outputs the preliminary speech vector, the speaker object information is embedded into the preliminary speech vector according to an attention mechanism, and the preliminary speech vector is processed according to the speaker object information to improve the weight of the vector corresponding to the speaker object information in the preliminary speech vector, to obtain a decoding candidate speech vector, so that the output decoding candidate speech vector is more consistent with the speech characteristics of the speaker.

[0135] In step S603 of some embodiments, the decoding candidate speech vector is input into a second decoder, and the second decoder decodes the decoding candidate speech vector to convert the vector-form decoding candidate speech vector into speech spectrum data in the form of a data sequence, so that the speech synthesis operation is simple.

[0136] In step S604 of some embodiments, the speech spectrum data is output by the second decoder to realize speech synthesis, and in order to directly obtain speech waveform data, the speech spectrum data is input into a vocoder to perform waveform calculation on the speech spectrum data by the vocoder to obtain speech synthesis prediction data, and the speech synthesis prediction data is waveform data based on mel spectrum, so that an audio player can directly play audio according to the speech synthesis prediction data.

[0137] Please refer to Figure 7 The embodiments of the present application also provide a speech synthesis method, which can include but is not limited to steps S701 to S703:

[0138] Step S701, obtaining target phoneme data;

[0139] Step S702, inputting the target phoneme data into a target speech synthesis model; wherein the target speech synthesis model is obtained by the model training method as described above;

[0140] Step S703, performing speech synthesis on the target phoneme data by the target speech synthesis model to obtain target speech synthesis data; wherein the target speech synthesis data is waveform data based on mel spectrum.

[0141] The steps S701 to S703 shown in the embodiments of the present application are to obtain target phoneme data, input the target phoneme data into a target speech synthesis model, perform speech synthesis on the target phoneme data by the target speech synthesis model to obtain target speech synthesis data, so that the speech synthesis operation is simple. The target speech synthesis model is obtained by the model training method described above, so as to construct an accurate target speech synthesis model in the scene of a small amount of sample resources, and the target speech synthesis data obtained by performing speech synthesis on the target phoneme data by the target speech synthesis model is also more accurate.

[0142] In step S701 of some embodiments, if target audio data is obtained, the target audio data is analyzed for audio content to obtain target audio text, and the target audio text is processed for word segmentation to obtain target phoneme data, so as to directly determine the target phoneme data of the user by directly speaking. If the target audio text is obtained, the target audio text is directly processed for word segmentation to obtain the target phoneme data, that is, speech synthesis can be realized by inputting the text, so as to expand the data type input during speech synthesis, thereby improving the experience of the user during speech synthesis.

[0143] In steps S702 and S703 of some embodiments, the target phoneme data is input into the target speech synthesis model, and the specific structure of the target speech synthesis model is consistent with that of the original speech synthesis model, that is, the target phoneme data is processed for encoding, speaker object information embedding, decoding, and waveform calculation by the target speech synthesis model to obtain target speech synthesis data, so that the speech synthesis operation is simple and the synthesis effect is better.

[0144] The embodiment of the application obtains sample speech feature data and sample phoneme data, inputs the sample speech feature into a first encoder for encoding processing to obtain a preliminary feature vector, obtains a codebook vector of a preset codebook through a network layer, maps the preliminary speech feature vector with the codebook vector to obtain an embedded candidate feature vector, embeds the speaker object information into the embedded candidate feature vector through the network layer to obtain a decoding candidate feature vector, and decodes the decoding candidate feature vector through a first decoder to obtain preliminary speech feature data. The feature loss value is obtained by loss calculation on the sample speech feature data and the preliminary speech feature data, and the parameter adjustment is performed on the original feature generation model according to the feature loss value, and the feature loss value calculation and parameter adjustment of the original feature generation model are continuously cycled until the feature loss value is minimum to obtain a target feature generation model. The sample speech feature data is input into the target feature generation model for feature expansion processing to obtain target speech feature data closer to the sample speech feature data. Therefore, the target speech feature data and the sample speech feature data are collected to obtain updated speech feature data, and the updated speech feature data and the sample phoneme data are filtered according to the preset data mapping relationship to filter out the updated speech feature data corresponding to the sample phoneme data to obtain audio pairing data, and the target sample data is constructed according to the audio pairing data, the updated speech feature data and the sample phoneme data to realize the expansion of the sample data. Therefore, the sample phoneme data is input into a second encoder for encoding processing to obtain a preliminary speech vector, the preliminary speech vector is processed by attention according to the preset speaker object information to obtain a decoding candidate speech vector, and the decoding candidate speech vector is decoded by a second decoder to obtain speech spectrum data; the speech spectrum data is calculated by a vocoder to obtain speech synthesis prediction data, and the updated speech feature data is filtered according to the audio pairing data and the sample phoneme data to obtain speech feature verification data, and finally the original speech synthesis model is adjusted according to the speech feature verification data and the speech synthesis prediction data to obtain a target speech synthesis model to construct a target speech synthesis model with better speech synthesis effect, and the target phoneme data is synthesized by the target speech synthesis model to obtain waveform data based on mel spectrum. Therefore, in the scene of a small amount of sample resources, a target speech synthesis model with better speech synthesis effect is constructed.

[0145] Please refer to Figure 8 The embodiment of the application also provides a model training device, which can realize the above-mentioned model training method, and the device comprises:

[0146] The sample acquisition module 801 is configured to acquire original sample data; wherein the original sample data comprises sample speech feature data and sample phoneme data.

[0147] The data input module 802 is configured to input sample voice feature data into a preset original feature generation model for feature expansion processing, to obtain preliminary voice feature data.

[0148] The loss calculation module 803 is configured to perform loss calculation on the sample voice feature data and the preliminary voice feature data, to obtain a feature loss value.

[0149] The parameter adjustment module 804 is configured to perform parameter adjustment on the original feature generation model according to the feature loss value, until the feature loss value is minimum, to obtain a target feature generation model.

[0150] The data expansion module 805 is configured to input the sample voice feature data into the target feature generation model for feature expansion processing, to obtain target voice feature data.

[0151] The data collection module 806 is configured to construct target sample data according to the sample voice feature data, the target voice feature data and sample phoneme data.

[0152] The model training module 807 is configured to perform model training on a preset original voice synthesis model according to the target sample data, to obtain a target voice synthesis model.

[0153] The specific implementation of the model training apparatus is basically the same as the specific embodiments of the model training method described above, and will not be repeated here.

[0154] Please refer to Figure 9 The embodiment of the present application also provides a voice synthesis apparatus, which can implement the voice synthesis method described above, and the apparatus comprises:

[0155] The target acquisition module 901 is configured to acquire target phoneme data.

[0156] The phoneme input module 902 is configured to input the target phoneme data into a target voice synthesis model; wherein the target voice synthesis model is obtained by the model training apparatus as described above.

[0157] The voice synthesis module 903 is configured to perform voice synthesis on the target phoneme data by the target voice synthesis model, to obtain target voice synthesis data; wherein the target voice synthesis data is waveform data based on mel-frequency spectrum.

[0158] The specific implementation of the voice synthesis apparatus is basically the same as the specific embodiments of the voice synthesis method described above, and will not be repeated here.

[0159] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the model training method or the voice synthesis method described above when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0160] Please refer to Figure 10 , Figure 10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device comprises:

[0161] The processor 101 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0162] The memory 102 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 102 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 102 and are called and executed by the processor 101 to implement the above-mentioned model training method or speech synthesis method of the embodiments of the present application.

[0163] The input / output interface 103 is configured to implement information input and output.

[0164] The communication interface 104 is configured to implement the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, USB, network cable, etc.) or a wireless manner (for example, mobile network, WIFI, Bluetooth, etc.).

[0165] The bus 105 is configured to transmit information between various components (for example, the processor 101, the memory 102, the input / output interface 103, and the communication interface 104) of the device.

[0166] The processor 101, the memory 102, the input / output interface 103, and the communication interface 104 are connected to each other through the bus 105 to realize the communication connection between them in the device.

[0167] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned model training method or speech synthesis method is realized.

[0168] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The model training method and device, the speech synthesis method and device, the equipment and the storage medium provided by the embodiments of the present application obtain sample speech feature data and sample phoneme data, input the sample speech feature data into a preset original feature generation model to perform feature expansion processing to obtain preliminary speech feature data, perform loss calculation on the sample speech feature data and the preliminary speech feature data to obtain a feature loss value, adjust parameters of the original feature generation model according to the feature loss value, and continuously loop the feature loss value calculation and parameter adjustment of the original feature generation model until the feature loss value is minimum to obtain a target feature generation model, then input the sample speech feature data into the target feature generation model to perform feature expansion processing to obtain target speech feature data, so as to obtain target speech feature data closer to the sample speech feature data, construct target sample data according to the target speech feature data, the sample speech feature data and the sample phoneme data to expand the sample data, and finally train an original speech synthesis model according to the expanded target sample data to obtain a target speech synthesis model, and the speech synthesis effect of the target speech synthesis model is better.

[0170] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0171] Those skilled in the art can understand that, Figures 1-7 The technical solutions shown in the above description do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figure, or combine certain steps, or different steps.

[0172] The device embodiments described above are only schematic, and units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0173] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0174] The terms "first", "second", "third", "fourth" and the like in the description of the application, and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a general order and / or structure unless otherwise indicated. Furthermore, the terms "comprise", "have" and "include" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has or includes a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0175] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0176] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0178] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0179] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0180] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining original sample data; wherein the original sample data comprises sample voice feature data and sample phoneme data; inputting the sample voice feature data into a preset original feature generation model for feature expansion processing to obtain preliminary voice feature data; calculating a loss of the sample voice feature data and the preliminary voice feature data to obtain a feature loss value; adjusting parameters of the original feature generation model according to the feature loss value until the feature loss value is minimum to obtain a target feature generation model; inputting the sample voice feature data into the target feature generation model for feature expansion processing to obtain target voice feature data; constructing target sample data according to the sample voice feature data, the target voice feature data and the sample phoneme data; training a preset original voice synthesis model according to the target sample data to obtain a target voice synthesis model; the original feature generation model comprises a first encoder, a first decoder and a network layer; the inputting the sample voice feature data into a preset original feature generation model for feature expansion processing to obtain preliminary voice feature data comprises: inputting the sample voice feature data into the first encoder for encoding processing to obtain a preliminary feature vector; transforming the preliminary feature vector by the network layer according to a preset codebook to obtain an embedded candidate feature vector; embedding preset speaker information into the embedded candidate feature vector by the network layer to obtain a decoded candidate feature vector; decoding the decoded candidate feature vector by the first decoder to obtain the preliminary voice feature data.

2. The method of claim 1, wherein, the transforming the preliminary feature vector by the network layer according to a preset codebook to obtain an embedded candidate feature vector comprises: obtaining a discrete variable of the preset codebook by the network layer to obtain a codebook vector; mapping the preliminary feature vector by the network layer according to the codebook vector to obtain the embedded candidate feature vector.

3. The method according to any one of claims 1 to 2, characterized in that, the constructing target sample data according to the sample voice feature data, the target voice feature data and the sample phoneme data comprises: merging the sample voice feature data and the target voice feature data to obtain updated voice feature data; screening the updated voice feature data and the sample phoneme data according to a preset data mapping relationship to obtain audio pairing data; constructing the target sample data according to the audio pairing data, the updated voice feature data and the sample phoneme data.

4. The method of claim 3, wherein, the training a preset original voice synthesis model according to the target sample data to obtain a target voice synthesis model comprises: inputting the sample phoneme data into the original voice synthesis model for voice synthesis to obtain voice synthesis prediction data; screening the updated voice feature data according to the audio pairing data and the sample phoneme data to obtain voice feature verification data; According to the voice feature verification data and the voice synthesis prediction data, the original voice synthesis model is adjusted in parameters to obtain the target voice synthesis model.

5. The method of claim 4, wherein, The original voice synthesis model comprises a second encoder, a second decoder and a vocoder; and the inputting of the sample phoneme data into the original voice synthesis model for voice synthesis to obtain voice synthesis prediction data comprises: The sample phoneme data is input into the second encoder for encoding processing to obtain a preliminary voice vector; The preliminary voice vector is subjected to attention processing according to preset speaker information to obtain a decoding candidate voice vector; The decoding candidate voice vector is subjected to decoding processing by the second decoder to obtain voice spectrum data; The voice spectrum data is subjected to waveform calculation by the vocoder to obtain the voice synthesis prediction data.

6. A speech synthesis method characterized by, The method comprises: obtaining target phoneme data; inputting the target phoneme data into a target voice synthesis model; wherein the target voice synthesis model is obtained by the model training method according to any one of claims 1 to 5; performing voice synthesis on the target phoneme data by the target voice synthesis model to obtain target voice synthesis data; wherein the target voice synthesis data is waveform data based on mel spectrum.

7. A model training apparatus characterized by comprising: The device comprises: a sample acquisition module configured to acquire original sample data; wherein the original sample data comprises sample voice feature data and sample phoneme data; a data input module configured to input the sample voice feature data into a preset original feature generation model for feature expansion processing to obtain preliminary voice feature data; wherein the original feature generation model comprises a first encoder, a first decoder and a network layer; and the inputting of the sample voice feature data into the preset original feature generation model for feature expansion processing to obtain preliminary voice feature data comprises: inputting the sample voice feature data into the first encoder for encoding processing to obtain a preliminary feature vector; transforming the preliminary feature vector by the network layer according to a preset codebook to obtain an embedded candidate feature vector; embedding preset speaker information into the embedded candidate feature vector by the network layer to obtain a decoding candidate feature vector; decoding the decoding candidate feature vector by the first decoder to obtain the preliminary voice feature data; a loss calculation module configured to calculate a feature loss value by performing loss calculation on the sample voice feature data and the preliminary voice feature data; a parameter adjustment module configured to adjust the original feature generation model in parameters according to the feature loss value until the feature loss value is minimum to obtain a target feature generation model; a data expansion module configured to input the sample voice feature data into the target feature generation model for feature expansion processing to obtain target voice feature data; a data collection module configured to construct target sample data according to the sample voice feature data, the target voice feature data and the sample phoneme data. A model training module configured to perform model training on a preset original speech synthesis model according to the target sample data to obtain a target speech synthesis model.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 5 or the method in claim 6 when executing the computer program.

9. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 5 or the method in claim 6. The computer program is executed by the processor to implement the method in any one of claims 1 to 5 or the method in claim 6.

Images