Speech synthesis model generation method based on rhythm boundary information and VAE structure

Abstract

The invention relates to a speech synthesis model generation method based on rhythm boundary information and a VAE structure. The method comprises the following steps: acquiring a training data set; the training data set comprises sample voice and rhythm boundary information, text information and phoneme information corresponding to the sample voice; training a speech synthesis network according to the sample speech and the corresponding phoneme information to obtain a trained speech synthesis network; according to the sample voice and the corresponding rhythm boundary information, training a rhythm coding network to obtain a trained rhythm coding network; according to the sample voice and the corresponding text information, phoneme information and rhythm boundary information, training a rhythm prediction network to obtain a trained rhythm prediction network; and according to the trained speech synthesis network, the trained rhythm coding network and the trained rhythm prediction network, generating a speech synthesis model based on rhythm boundary information and a VAE structure. By adopting the method, the synthetic speech with accurate rhythm can be generated, and the accuracy of the synthetic speech is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for generating a speech synthesis model based on prosodic boundary information and VAE structure. Background Art

[0002] With the development of information technology and deep learning technology, human-computer interaction has become an important technology to enhance user experience in education, entertainment, security and other fields. Among the many human-computer interaction methods, voice, with its intuitive and efficient information interaction advantages, has become the most commonly used human-computer interaction method in the field of human-computer interaction. Voice interaction generally includes voice understanding and voice output, among which voice output is achieved by speech synthesis technology. Speech synthesis is the technology of text-to-speech (TTS), which can convert the text information input into the speech synthesis system into human-understandable and natural voice information output.

[0003] Traditional technology uses parameter synthesis and waveform splicing as the main means of speech synthesis. It builds a speech database in advance, selects corresponding speech data from the speech database in combination with input text and corresponding rules, and splices the speech data to achieve speech synthesis. However, the accuracy of the synthesized speech obtained by traditional technology depends on the speech synthesis rules, and the sentence intonation of the synthesized speech is flat and lacks changes, and cannot reflect the accurate speech rhythm. Therefore, traditional technology has the problem of low accuracy of synthesized speech. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for generating a speech synthesis model based on prosodic boundary information and VAE structure, which can improve the accuracy of synthesized speech in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for generating a speech synthesis model based on prosodic boundary information and a VAE structure, comprising:

[0006] Acquire a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech;

[0007] Training a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network;

[0008] Training a prosody coding network according to the sample speech and prosody boundary information corresponding to the sample speech to obtain a trained prosody coding network;

[0009] Training a prosody prediction network according to the sample speech and text information, phoneme information, and prosody boundary information corresponding to the sample speech to obtain a trained prosody prediction network;

[0010] The speech synthesis model based on prosodic boundary information and VAE structure is generated according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

[0011] In one embodiment, the training of a prosody prediction network according to the sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech includes:

[0012] Inputting the phoneme information corresponding to the sample speech into the trained speech synthesis network to obtain the phoneme feature information corresponding to the phoneme information;

[0013] Inputting the phoneme feature information and the text information corresponding to the sample speech into the prosody prediction network to obtain a prosody prediction result;

[0014] Inputting the sample speech and the prosodic boundary information corresponding to the sample speech into the trained prosodic coding network to obtain prosodic feature information;

[0015] The rhythm prediction network is trained according to the rhythm prediction result and the rhythm feature information.

[0016] In one embodiment, the training of the prosody prediction network according to the prosody prediction result and the prosody feature information includes:

[0017] Filtering candidate prosody feature information from the prosody feature information;

[0018] The prosody prediction network is trained according to the prosody prediction result and the candidate prosody feature information.

[0019] In one embodiment, the training of a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech includes:

[0020] Inputting the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network to obtain latent variable feature information corresponding to the sample speech;

[0021] Generate a speech synthesis result corresponding to the sample speech according to the latent variable feature information through the speech synthesis network;

[0022] The speech synthesis network is trained according to the speech synthesis result and the sample speech.

[0023] In one embodiment, the training of a prosody coding network according to the sample speech and the prosody boundary information corresponding to the sample speech includes:

[0024] Inputting the phoneme information corresponding to the sample speech into the speech synthesis network to obtain a speech synthesis result corresponding to the sample speech;

[0025] Inputting the speech synthesis result into the prosody coding network to obtain prosody feature information corresponding to the speech synthesis result;

[0026] The prosodic coding network is trained according to the prosodic feature information corresponding to the speech synthesis result and the prosodic boundary information corresponding to the sample speech.

[0027] In a second aspect, the present application also provides a speech generation method of a speech synthesis model based on prosodic boundary information and a VAE structure, comprising:

[0028] Inputting text information corresponding to the speech to be generated into a prosody prediction network in a pre-trained speech synthesis model based on prosody boundary information and VAE structure to obtain a prosody prediction result corresponding to the speech to be generated;

[0029] The phoneme information corresponding to the speech to be generated and the prosody prediction result corresponding to the speech to be generated are input into the speech synthesis network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure to generate the speech to be generated.

[0030] In a third aspect, the present application further provides a speech synthesis model generation device based on prosodic boundary information and VAE structure, comprising:

[0031] A training data acquisition module, used to acquire a training data set; the training data set includes a sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech;

[0032] A synthesis network training module, used for training a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network;

[0033] A coding network training module, used for training a prosodic coding network according to the sample speech and the prosodic boundary information corresponding to the sample speech, to obtain a trained prosodic coding network;

[0034] A prediction network training module, used for training a prosody prediction network according to the sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech, to obtain a trained prosody prediction network;

[0035] A model generation module is used to generate the speech synthesis model based on prosodic boundary information and VAE structure according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

[0036] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0037] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0038] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0039] The above-mentioned speech synthesis model generation method, device, computer equipment, computer-readable storage medium and computer program product based on prosodic boundary information and VAE structure obtain a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech, and trains a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network, so that the sample speech in the training data set and the phoneme information corresponding to the sample speech can be used to train an accurate speech synthesis network and optimize the performance of the speech synthesis network; according to the sample speech and the prosodic boundary information corresponding to the sample speech, train a prosodic coding network to obtain a trained prosodic coding network, so that the sample speech in the training data set and the prosodic boundary information corresponding to the sample speech can be used to train an accurate prosodic coding network and optimize the performance of the prosodic coding network, improve the accuracy of the prosodic information generated by the prosodic coding network, and facilitate the speech synthesis network to combine the prosodic information generated by the prosodic coding network to generate a synthetic speech with rhythm. Speech; training a prosody prediction network according to sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech to obtain a trained prosody prediction network, thereby using the sample speech in the training data set and the text information, phoneme information and prosody boundary information corresponding to the sample speech to train an accurate prosody prediction network, optimize the performance of the prosody prediction network, improve the accuracy of the prosody prediction results generated by the prosody prediction network, and facilitate the prosody coding network to generate accurate prosody information in combination with the prosody prediction results generated by the prosody prediction network; generating a speech synthesis model based on prosody boundary information and VAE structure according to the trained speech synthesis network, the trained prosody coding network and the trained prosody prediction network, thereby respectively training the speech synthesis network, the prosody coding network and the prosody prediction network based on the training data set, so that the prosody coding network can generate accurate prosody information in combination with the prediction results of the prosody prediction network, and the speech synthesis network can generate synthesized speech with accurate prosody in combination with the prosody information generated by the prosody coding network, thereby improving the accuracy of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 An application environment diagram of a method for generating a speech synthesis model based on prosodic boundary information and a VAE structure in one embodiment;

[0042] Figure 2 A flowchart of a method for generating a speech synthesis model based on prosodic boundary information and a VAE structure in one embodiment;

[0043] Figure 3 is a schematic diagram of the structure of a speech synthesis network in one embodiment;

[0044] Figure 4 is a schematic diagram of the structure of a feedforward neural network in one embodiment;

[0045] Figure 5 is a schematic diagram of a training speech synthesis network in one embodiment;

[0046] Figure 6 A schematic diagram of the structure of a speech synthesis model based on prosodic boundary information and VAE structure in one embodiment;

[0047] Figure 7 is a schematic diagram of the structure of a prosody coding network in one embodiment;

[0048] Figure 8 A schematic diagram of the structure of a rhythm prediction network in one embodiment;

[0049] Fig. 9 A flowchart of a speech synthesis method based on a speech synthesis model of prosody boundary information and a VAE structure in one embodiment;

[0050] Fig.10 It is a structural block diagram of a speech synthesis model generation device based on prosody boundary information and VAE structure in one embodiment;

[0051] Fig.11 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The speech synthesis model generation method based on prosodic boundary information and VAE structure provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech; the server 104 trains the speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain the trained speech synthesis network; the server 104 trains the prosodic coding network according to the sample speech and the prosodic boundary information corresponding to the sample speech to obtain the trained prosodic coding network; the server 104 trains the prosodic prediction network according to the sample speech and the text information, phoneme information and prosodic boundary information corresponding to the sample speech to obtain the trained prosodic prediction network; the server 104 generates a speech synthesis model based on prosodic boundary information and VAE structure according to the trained speech synthesis network, the trained prosodic coding network and the trained prosodic prediction network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a method for generating a speech synthesis model based on prosodic boundary information and VAE structure is provided, and the method is applied to a server as an example for explanation, including the following steps S202 to S210. Among them:

[0055] Step S202, obtaining a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech.

[0056] Among them, the training data set may refer to a set of data composed of speech synthesis networks, prosody encoding networks and prosody prediction networks in speech synthesis models based on prosody boundary information and VAE structure. In practical applications, the training data set may include but is not limited to sample speech, prosody boundary information corresponding to the sample speech, text information corresponding to the sample speech and phoneme information corresponding to the sample speech. Among them, the prosody boundary information may refer to information describing the rhythm of the sample speech. In a specific implementation, the prosody boundary information may include prosody boundary label information, the text information may refer to information describing the content of the sample speech, and the text information may include a text sequence. The phoneme information may refer to information describing the phonemes corresponding to the sample speech, and the phoneme information may include a phoneme sequence.

[0057] As an example, in order to train a speech synthesis model based on prosodic boundary information and VAE structure, the server can pre-build a sample database, and when the speech synthesis model based on prosodic boundary information and VAE structure needs to be trained, obtain some data from the sample database as a training data set, the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech for training the speech synthesis model.

[0058] Step S204: training a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network.

[0059] Among them, the speech synthesis network can refer to a model that uses text and / or phonemes to generate corresponding speech. In practical applications, the speech synthesis network can include a non-autoregressive end-to-end speech synthesis acoustic model (such as the FastSpeech model), and the speech output by the speech synthesis network can be used as synthesized speech.

[0060] As an example, the server inputs the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network to be trained. The speech synthesis network can use the phoneme information to generate the corresponding speech as the output result. The server can adjust the network parameters of the speech synthesis network based on the difference between the output result of the speech synthesis network and the sample speech until the difference between the output result of the speech synthesis network and the sample speech meets the preset speech requirements. The server uses the speech synthesis network at this time as the trained speech synthesis network.

[0061] Step S206: training a prosodic coding network according to the sample speech and the prosodic boundary information corresponding to the sample speech to obtain a trained prosodic coding network.

[0062] The prosody coding network may refer to a model used to obtain the output result of the speech synthesis network and / or the prosody boundary information of the sample speech. In practical applications, the prosody boundary information output by the prosody coding network may be used as prosody feature information.

[0063] As an example, the server inputs the sample speech and the prosodic boundary information corresponding to the sample speech into the prosodic coding network to be trained. The prosodic coding network can analyze the prosodic features of the sample speech and generate prosodic feature information. The server can adjust the network parameters of the prosodic coding network based on the difference between the prosodic feature information output by the prosodic coding network and the prosodic boundary information corresponding to the sample speech until the difference between the prosodic feature information output by the prosodic coding network and the prosodic boundary information corresponding to the sample speech meets the preset prosodic difference requirement. The server uses the prosodic coding network at this time as the trained prosodic coding network.

[0064] Step S208: training a prosody prediction network according to the sample speech and the text information, phoneme information and prosody boundary information corresponding to the sample speech to obtain a trained prosody prediction network.

[0065] The prosody prediction network may refer to a model for generating prosody boundary information of a sample speech using text information and phoneme information corresponding to the sample speech. In practical applications, the prosody prediction result output by the prosody prediction network may be used as the prosody prediction result.

[0066] As an example, the server inputs the text information and phoneme information corresponding to the sample speech into the prosody prediction network to be trained. The prosody prediction network can analyze the text information and phoneme information corresponding to the sample speech to generate a prosody prediction result. The server can adjust the network parameters of the prosody prediction network according to the difference between the prosody prediction result and the prosody boundary information corresponding to the sample speech, until the difference between the prosody prediction result and the prosody boundary information corresponding to the sample speech meets the preset prediction difference requirement, and the server uses the prosody prediction network at this time as the trained prosody prediction network. The server can also adjust the network parameters of the prosody prediction network according to the prosody feature information output by the prosody encoding network, the prosody prediction result generated by the prosody prediction network, and the prosody boundary information corresponding to the sample speech after the prosody prediction network generates the prosody prediction result, until the difference between the prosody prediction result and the prosody boundary information corresponding to the sample speech meets the preset prediction difference requirement, and the server uses the prosody prediction network at this time as the trained prosody prediction network.

[0067] Step S210, generating a speech synthesis model based on prosodic boundary information and VAE structure according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

[0068] As an example, the server can construct a speech synthesis model based on prosodic boundary information and VAE structure according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network. The server can also use the training data set to further train and verify the speech synthesis model based on prosodic boundary information and VAE structure, thereby optimizing the model performance of the speech synthesis model based on prosodic boundary information and VAE structure, and ensuring that the synthesized speech has accurate rhythm.

[0069] In the above-mentioned method for generating a speech synthesis model based on prosodic boundary information and VAE structure, a training data set is obtained; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech, and a speech synthesis network is trained according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network, so that the sample speech in the training data set and the phoneme information corresponding to the sample speech can be used to train an accurate speech synthesis network and optimize the performance of the speech synthesis network; a prosodic coding network is trained according to the sample speech and the prosodic boundary information corresponding to the sample speech to obtain an accurate prosodic coding network and optimize the performance of the prosodic coding network, improve the accuracy of the prosodic information generated by the prosodic coding network, and facilitate the speech synthesis network to generate rhythmic synthetic speech in combination with the prosodic information generated by the prosodic coding network; according to the sample speech and the sample speech The text information, phoneme information and prosodic boundary information corresponding to the speech are used to train the prosodic prediction network to obtain the trained prosodic prediction network, thereby utilizing the sample speech in the training data set and the text information, phoneme information and prosodic boundary information corresponding to the sample speech to train an accurate prosodic prediction network, optimize the performance of the prosodic prediction network, improve the accuracy of the prosodic prediction results generated by the prosodic prediction network, and facilitate the prosodic coding network to generate accurate prosodic information in combination with the prosodic prediction results generated by the prosodic prediction network; according to the trained speech synthesis network, the trained prosodic coding network and the trained prosodic prediction network, a speech synthesis model based on prosodic boundary information and VAE structure is generated, thereby respectively training the speech synthesis network, the prosodic coding network and the prosodic prediction network based on the training data set, so that the prosodic coding network can generate accurate prosodic information in combination with the prediction results of the prosodic prediction network, and the speech synthesis network can generate synthesized speech with accurate prosody in combination with the prosodic information generated by the prosodic coding network, thereby improving the accuracy of the synthesized speech.

[0070] In an exemplary embodiment, a prosody prediction network is trained based on sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech, including: inputting the phoneme information corresponding to the sample speech into a trained speech synthesis network to obtain phoneme feature information corresponding to the phoneme information; inputting the phoneme feature information and text information corresponding to the sample speech into the prosody prediction network to obtain a prosody prediction result; inputting the sample speech and the prosody boundary information corresponding to the sample speech into a trained prosody encoding network to obtain prosody feature information; and training the prosody prediction network based on the prosody prediction result and the prosody feature information.

[0071] The phoneme feature information may refer to information obtained by encoding the phoneme information of the sample speech.

[0072] The prosody prediction result may refer to information representing the prosody obtained after the prosody prediction network analyzes the phoneme feature information of the phoneme information corresponding to the sample speech and the text information corresponding to the sample speech.

[0073] The prosody feature information may refer to information representing the prosody obtained after the prosody coding network analyzes the sample speech and the prosody boundary information corresponding to the sample speech.

[0074] As an example, the server inputs the phoneme information corresponding to the sample speech into the trained speech synthesis network. The trained speech synthesis network can encode the phoneme information to obtain the phoneme feature information corresponding to the phoneme information. Then the server can input the phoneme feature information corresponding to the phoneme information and the text information corresponding to the sample speech into the prosody prediction network. The prosody prediction network can analyze the phoneme feature information and the text information to generate a prosody prediction result. The server can input the sample speech and the prosody boundary information corresponding to the sample speech into the trained prosody coding network. The trained prosody coding network can analyze the sample speech and the prosody boundary information corresponding to the sample speech to generate prosody feature information. The server can use the prosody feature information and / or the prosody boundary information corresponding to the sample speech as a reference to analyze the difference between the prosody feature information and / or the prosody boundary information corresponding to the sample speech and the prosody prediction result output by the prosody prediction network, and adjust the prosody prediction network until the difference between the prosody prediction result and the prosody feature information and / or the prosody boundary information corresponding to the sample speech meets the preset prediction difference requirement. The server uses the prosody prediction network at this time as the trained prosody prediction network.

[0075] In this embodiment, the phoneme information corresponding to the sample speech is input into the trained speech synthesis network to obtain the phoneme feature information corresponding to the phoneme information; the phoneme feature information and the text information corresponding to the sample speech are input into the prosody prediction network to obtain the prosody prediction result; the sample speech and the prosody boundary information corresponding to the sample speech are input into the trained prosody coding network to obtain the prosody feature information; according to the prosody prediction result and the prosody feature information, the prosody prediction network is trained, and the prosody prediction result can be generated based on the phoneme feature information generated by the trained speech synthesis network and the text information corresponding to the sample speech, and the prosody prediction network is trained in combination with the prosody feature information output by the trained prosody coding network to optimize the performance of the prosody prediction network and improve the accuracy of the prosody prediction result output by the prosody prediction network, so as to provide a data basis for the speech synthesis network to generate a synthesized speech with accurate prosody in combination with the prosody prediction result output by the prosody prediction network, thereby improving the accuracy of the synthesized speech.

[0076] In some embodiments, training a prosody prediction network according to the prosody prediction results and the prosody feature information includes: selecting candidate prosody feature information from the prosody feature information; and training a prosody prediction network according to the prosody prediction results and the candidate prosody feature information.

[0077] The candidate prosodic feature information may refer to information screened out from the prosodic feature information according to a preset condition. In practical applications, the candidate prosodic feature information may refer to information screened out from the prosodic feature information according to a preset ratio.

[0078] As an example, during the process of training the rhythm prediction network, the server may not use all the rhythm feature information to train the rhythm prediction network, but may screen out candidate rhythm feature information from the rhythm feature information at a preset ratio, and use the rhythm prediction results and the candidate rhythm feature information to train the rhythm prediction network, thereby reducing the difference between the training and application processes / inference processes, and avoiding the mismatch problem between training and reasoning caused by the gap between the rhythm feature information predicted in the reasoning stage and the real counterpart (such as rhythm boundary information).

[0079] In this embodiment, by screening out candidate prosody feature information from the prosody feature information; and training a prosody prediction network based on the prosody prediction results and the candidate prosody feature information, it is possible to avoid the prosody prediction network from relying on all prosody feature information for training, thereby reducing the difference between the training process and the reasoning process of the prosody prediction network, and further improving the accuracy of the prosody prediction results output by the prosody prediction network, providing a data basis for the speech synthesis network to generate synthesized speech with accurate prosody in combination with the prosody prediction results output by the prosody prediction network, thereby improving the accuracy of the synthesized speech.

[0080] In some embodiments, a speech synthesis network is trained based on sample speech and phoneme information corresponding to the sample speech, including: inputting the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network to obtain latent variable feature information corresponding to the sample speech; generating a speech synthesis result corresponding to the sample speech based on the latent variable feature information through the speech synthesis network; and training the speech synthesis network based on the speech synthesis result and the sample speech.

[0081] The latent variable feature information may refer to information characterizing the features of the sample speech in a preset space (such as a latent space). In practical applications, the latent variable feature information may include latent variables.

[0082] The speech synthesis result may refer to the synthesized speech generated by the speech synthesis network based on the phoneme information corresponding to the sample speech.

[0083] As an example, the server inputs the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network. The speech synthesis network can encode the phoneme information corresponding to the sample speech to obtain phoneme feature information, and convert the phoneme feature information to a preset space to obtain latent variable feature information of the phoneme information corresponding to the sample speech. The speech synthesis network can decode the latent variable feature information to obtain a speech synthesis result. The server can analyze the difference between the speech synthesis result and the sample speech, and adjust the network parameters of the speech synthesis network until the difference between the speech synthesis result and the sample speech meets the preset speech requirements. The server uses the speech synthesis network at this time as the trained speech synthesis network.

[0084] In this embodiment, the sample speech and the phoneme information corresponding to the sample speech are input into the speech synthesis network to obtain the latent variable feature information corresponding to the sample speech; the speech synthesis result corresponding to the sample speech is generated by the speech synthesis network according to the latent variable feature information; the speech synthesis network is trained according to the speech synthesis result and the sample speech, so that the phoneme information corresponding to the sample speech can be converted into latent variable feature information, and the speech synthesis network is trained in combination with the difference between the speech synthesis result generated by the speech synthesis network based on the latent variable feature information and the sample speech, thereby improving the accuracy of the synthesized speech.

[0085] In some embodiments, a prosody coding network is trained based on sample speech and prosody boundary information corresponding to the sample speech, including: inputting phoneme information corresponding to the sample speech into a speech synthesis network to obtain a speech synthesis result corresponding to the sample speech; inputting the speech synthesis result into a prosody coding network to obtain prosody feature information corresponding to the speech synthesis result; and training the prosody coding network based on the prosody feature information corresponding to the speech synthesis result and the prosody boundary information corresponding to the sample speech.

[0086] As an example, the server inputs the phoneme information corresponding to the sample speech into the speech synthesis network. The speech synthesis network can encode the phoneme information corresponding to the sample speech and convert the encoding result into latent variable feature information. The latent variable feature information is then used to generate a speech synthesis result corresponding to the phoneme information of the sample speech. The server inputs the speech synthesis result into the prosody coding network. The prosody coding network can analyze the speech synthesis result and generate prosody feature information corresponding to the speech synthesis result. The server can then adjust the network parameters of the prosody coding network according to the difference between the prosody feature information corresponding to the speech synthesis result and the prosody boundary information corresponding to the sample speech until the difference between the prosody feature information corresponding to the speech synthesis result and the prosody boundary information corresponding to the sample speech meets the preset prosody difference requirement. The server uses the prosody coding network at this time as the trained prosody coding network.

[0087] In this embodiment, the phoneme information corresponding to the sample speech is input into the speech synthesis network to obtain the speech synthesis result corresponding to the sample speech; the speech synthesis result is input into the prosody coding network to obtain the prosody feature information corresponding to the speech synthesis result; the prosody coding network is trained according to the prosody feature information corresponding to the speech synthesis result and the prosody boundary information corresponding to the sample speech, and the prosody coding network can be used to analyze the difference between the prosody feature information obtained from the speech synthesis result and the prosody boundary information corresponding to the sample speech, so as to train the prosody coding network, thereby improving the accuracy of the prosody feature information output by the prosody coding network, providing a data basis for the subsequent speech synthesis network to generate synthesized speech in combination with the prosody feature information output by the prosody coding network, thereby improving the accuracy of the synthesized speech.

[0088] In some embodiments, a speech generation method for a speech synthesis model based on prosodic boundary information and a VAE structure is provided, comprising: inputting text information corresponding to the speech to be generated into a prosodic prediction network in a pre-trained speech synthesis model based on prosodic boundary information and a VAE structure to obtain a prosodic prediction result corresponding to the speech to be generated; inputting phoneme information corresponding to the speech to be generated and the prosodic prediction result corresponding to the speech to be generated into a speech synthesis network in a pre-trained speech synthesis model based on prosodic boundary information and a VAE structure to generate the speech to be generated.

[0089] As an example, the server inputs the phoneme information corresponding to the speech to be generated into a speech synthesis network in a pre-trained speech synthesis model based on prosodic boundary information and VAE structure. The speech synthesis network can encode the phoneme information corresponding to the speech to be generated to obtain the phoneme feature information of the speech to be generated. The server inputs the text information corresponding to the speech to be generated and the phoneme feature information of the speech to be generated into a prosodic prediction network in a pre-trained speech synthesis model based on prosodic boundary information and VAE structure. The prosodic prediction network analyzes the text information corresponding to the speech to be generated and the phoneme feature information of the speech to be generated to generate a prosodic prediction result corresponding to the speech to be generated. The speech synthesis network can use the phoneme feature information of the speech to be generated and the prosodic prediction result corresponding to the speech to be generated to generate a latent variable feature information corresponding to the speech to be generated. The speech synthesis network can decode the latent variable feature information corresponding to the speech to be generated to obtain the speech to be generated. Furthermore, the prosody encoding network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure can analyze the speech to be generated and generate the prosody feature information corresponding to the speech to be generated. The pre-trained speech synthesis model based on prosody boundary information and VAE structure can perform splicing and other processing on the prosody feature information corresponding to the speech to be generated and the prosody prediction result corresponding to the speech to be generated to obtain prosody embedding feature information. Then, the speech synthesis network uses the phoneme feature information and prosody embedding feature information of the speech to be generated to generate new latent variable feature information corresponding to the speech to be generated. The speech synthesis network can decode the new latent variable feature information to obtain new speech to be generated.

[0090] In this embodiment, the prosody prediction result corresponding to the speech to be generated is obtained by inputting the text information corresponding to the speech to be generated into the prosody prediction network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure; the phoneme information corresponding to the speech to be generated and the prosody prediction result corresponding to the speech to be generated are input into the speech synthesis network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure to generate the speech to be generated. The pre-trained speech synthesis model based on prosody boundary information and VAE structure can generate synthesized speech with accurate prosody in combination with the text information and phoneme information corresponding to the speech to be generated, thereby improving the accuracy of the synthesized speech.

[0091] In some embodiments, the speech synthesis network may include a non-autoregressive acoustic model. The non-autoregressive structure enables the speech synthesis network to have a significant improvement in the inference speed compared with the traditional acoustic model. In addition, since the phoneme-level feature expansion method is used to complete the mapping of the phoneme sequence to the Mel-spectrogram sequence length in a non-autoregressive manner, it not only accelerates the inference speed of the model, but also reduces the problems of missed pronunciations and repeated pronunciations caused by the autoregressive model relying on the attention mechanism to learn the alignment of phonemes and spectra, thereby improving the robustness of the acoustic model. Figure 3As shown, a structural schematic diagram of a speech synthesis network is provided. The speech synthesis network can be a sequence-to-sequence (Seq2Seq) model, which adopts an encoder-decoder structure. The basic module of the speech synthesis network is a feed-forward neural module (Feed-Forward Transformer, FFT). The encoder and decoder of the speech synthesis network are composed of a plurality of feed-forward neural modules superimposed. The speech synthesis network also includes a length regulator and a duration predictor to model information such as phoneme duration. The linear layer in the speech synthesis network is used to output the generated Mel spectrum features, and then generate synthetic speech based on the Mel spectrum features. The phoneme sequence is processed by the phoneme embedding layer of the speech synthesis network. The output result of the phoneme embedding layer is combined with the position coding, and enters the decoder after being processed by the feed-forward neural network and the length regulator of the encoder. The feed-forward neural network of the decoder processes the output result of the encoder in combination with the position coding, and finally generates Mel spectrum features through the linear output layer, and then generates synthetic speech.

[0092] In some embodiments, Figure 4As shown in FIG. 1 , a schematic diagram of the structure of a feedforward neural network is provided. The feedforward neural network is a feedforward structure composed of a multi-head attention network and a one-dimensional convolution. In the speech synthesis network, multiple FFT blocks are stacked to form an encoder and a decoder (expanded to the frame level). The encoder encodes the input phoneme sequence, and the decoder can decode the output of the encoder, thereby completing the conversion from text information to Mel spectrum. The encoder on the phoneme side has N repeated FFT blocks, and the decoder on the Mel spectrum side also has N FFT blocks. The length regulator is located between the encoder and the decoder to complete the mapping alignment of the phoneme sequence to the Mel spectrum sequence length, that is, to achieve the amplification of the phoneme-level features, so that the one-to-many generation mapping can be completed without using autoregression. The duration predictor in the length regulator realizes the prediction of the duration of each input phoneme and provides duration information for the length regulator. The self-attention network extracts input feature information by multi-head attention. The FFT module in the speech synthesis network also contains two one-dimensional convolutional networks with ReLU activation functions in addition to the attention layer. This convolutional structure can better utilize the neighborhood information of speech. At the same time, residual connections, layer normalization, and dropout layers are added after the self-attention network and the one-dimensional convolutional network. The length regulator can solve the problem of inconsistency between the length of the phoneme sequence and the Mel spectrum in speech synthesis. In human speech, each phoneme has a corresponding phoneme duration, which is generally between a few frames and more than ten frames. If the phoneme duration of a phoneme is n, it means that this phoneme corresponds to the Mel spectrum of these n frames, so the speech synthesis network uses the length regulator to directly copy the phoneme-level features to the frame level n times according to the duration of the phoneme, so that by performing the same copy operation on each phoneme, the frame-level features of the same length as the Mel spectrum can be obtained. At the same time, the interruptions between words, phrases, and sentences can be changed by directly modifying the duration of pauses and silences in the synthesized speech, thereby achieving the purpose of changing the rhythmic expression of the final synthesized speech.

[0093] In some embodiments, Figure 5As shown in FIG. 1 , a schematic diagram of training a speech synthesis network is provided. The speech synthesis network can have a model structure of a variational autoencoder. A variational autoencoder is a probabilistic model based on variational inference. The variational autoencoder and the autoencoder have similar encoding-decoding structures and belong to unsupervised generation models. The difference is that the autoencoder is a simple compression encoding, while in variational inference, in addition to the known data (such as observation data, training data, etc.), there is also a hidden variable z (hidden variable feature information / hidden variable). The variational autoencoder assumes that the distribution of the input data set x (display variable) is completely controlled by a set of hidden variables z, and this set of hidden variables is independent of each other and obeys the Gaussian distribution. The variational autoencoder allows the encoder to learn the hidden variable model of the input data, that is, to learn the parameter mean and variance of the Gaussian probability distribution of this set of hidden variables, and the hidden variable z can be sampled from the normal distribution of this set of distribution parameters, and then the decoder decodes the z hidden variable to reconstruct the input, which essentially realizes a continuous and smooth latent space representation. These hidden variables (latent variables) contain useful information about the type of output that the model needs to generate. The probability distribution of the latent variable z is represented by p(z). The variational autoencoder chooses the Gaussian distribution as a prior to learn the distribution p(z) so that it is easier to sample new samples during the inference phase. The desired goal of the variational autoencoder is to define a generative model containing the latent variable z, and maximize the likelihood of the distribution p(x) of the X variable under the parameters θ obtained through training. At this time, p(x) can be expressed as p θ (x), p θ The expression of (x) can be expressed as:

[0094] .

[0095] But since it is impossible to traverse the hidden variable z, p θ (x) is difficult to solve, and the variational autoencoder introduces a new probability distribution q φ (z|x), and try to get the lower bound of the log-likelihood function, and then maximize the lower bound, which is equivalent to approximately maximizing the log-likelihood function. At this time, we have:

[0096] .

[0097] The above formula is ultimately composed of three terms. The first two terms are calculable, but the third term is not. However, according to the properties of KL divergence, the third term must be greater than or equal to 0 (and by fitting the encoder neural network, it can be guaranteed to be 0 as much as possible, so that the first two terms are as close to the target likelihood function as possible), so we can get:

[0098] .

[0099] The variational autoencoder calls the right side of the above inequality a variational lower bound (ELBO), so the goal of the variational autoencoder is to maximize the variational lower bound, that is, to use the variational lower bound as the loss function of the model.

[0100] As a neural network model, the variational autoencoder uses the back-propagation algorithm to train the network. However, since the sampling operation is not differentiable, the variational autoencoder introduces the reparameterization trick to solve the parameter transmission problem of model training. For the distribution N (μ, σ) of the latent variable z, first sample the standard normal distribution N (0, 1), then consider the mean and variance parameters learned by the encoder, and let z = μ + σ * τ. In this way, the normal transmission of the gradient can be completed, thereby completing the training of the variational autoencoder model.

[0101] In some embodiments, Figure 6As shown, a structural schematic diagram of a speech synthesis model based on prosodic boundary information and VAE structure is provided. The speech synthesis model based on prosodic boundary information and VAE structure is a non-autoregressive model, which uses phoneme sequence and text sequence as input to predict mel spectrum features. The speech synthesis model based on prosodic boundary information and VAE structure includes a speech synthesis network, a prosodic encoding network and a prosodic prediction network. The speech synthesis model based on prosodic boundary information and VAE structure is based on the encoder-decoder structure and variance adaptation module of the speech synthesis network. The prosodic boundary is used as the time resolution, and a prosodic modeling network at the prosodic word level and the prosodic phrase level can be introduced, as well as a prosodic prediction network trained considering the lack of reference speech in the inference stage. Among them, the variance adaptation module performs supervised training, and the training data includes variance information such as phoneme-level fundamental frequency, phoneme-level pitch, and duration, so as to assist in completing the one-to-many mapping between text and speech and provide speech prosodic change information missing from the text. During the training process, the phoneme-level encoder converts the phoneme sequence into its hidden representation (such as latent variables), the character-level encoder completes the encoding of the input Chinese character sequence, the variance adaptation module predicts the variance information (including phoneme-level fundamental frequency, phoneme-level pitch, duration, etc.), and the prosody coding network provides prosody modeling encoding related to the prosody boundary level. Finally, the decoder inputs the frame-level encoded latent variables with variance information, prosody information (such as multi-layer prosody embedding vectors) and phoneme information, thereby completing the prediction of the frame-level Mel-spectrum. In the inference stage, the prosody coding network will be predicted by the prosody prediction module using the phoneme-level encoding information and the character-level encoding information as input. Prosody modeling is not only prosody modeling for the prosody word level and prosody phrase level, but also refers to the hierarchical dependency modeling of the prosody boundary structure. In the encoding stage, the model first extracts the prosody word-level Mel-spectrogram features from the Mel-spectrogram, and then uses the prosody word-level prosody encoder to obtain the posterior distribution of the data, and obtains the prosody word-level prosody embedding vector (hereinafter referred to as the prosody word embedding vector) by random sampling. Then, the prosodic phrase-level prosodic encoder refers to the characteristic that high-level prosodic features are built on the basis of the low-level ones in the prosodic boundary structure, and adds the prosodic word embedding vector as additional information input in addition to the Mel-spectrogram features at the prosodic phrase scale.

[0102] In some embodiments, the prosody modeling network has two parts: a prosody encoding network and a prosody prediction network. Figure 7As shown, a structural schematic diagram of a prosody coding network is provided. The (multi-level) prosody coding network can be composed of a prosody word-level prosody encoder and a prosody phrase-level prosody encoder. The prosody encoder is composed of two one-dimensional convolutional layers and a bidirectional gated recurrent unit (GRU) layer. The Mel-spectrogram of speech is divided into prosody word boundaries and prosody phrase boundaries. The divided prosody word-level / prosody phrase-level Mel-spectrograms are respectively passed through the corresponding prosody encoders, and finally the multi-level prosody embedding vectors at the final phoneme level are amplified and concatenated and fused with the phoneme coding vector, where the prosody word and prosody phrase boundary information is obtained by combining the results of the front-end prosody prediction with the forced aligned phoneme boundary information. The (multi-level) prosody prediction network includes a prosody word-level prosody predictor and a prosody phrase-level prosody predictor, such as Figure 8As shown in the figure, a schematic diagram of the structure of the prosody prediction network is provided. In the inference stage, since the model cannot obtain the reference speech corresponding to the required text, it is impossible to use the prosody encoder to obtain the multi-level prosody embedding. Therefore, it is necessary to use the prosody predictor to complete the prediction of the multi-level prosody embedding from the text information. It is composed of a GRU layer and a one-dimensional convolution layer. At the same time, referring to the idea of ​​BERT-Tacotron2, the BERT encoder is additionally introduced to provide character-level feature information in addition to the phoneme sequence, so as to improve the prediction ability of this part. For the prosody word-level prosody predictor, its prediction is based on the phoneme sequence information and the text sequence information. The phoneme-level information input uses the phoneme-level encoder of the acoustic model body, and the structure is the same as FastSpeech2, while the text sequence is the character-level feature information. The pre-trained BERT model is used as the character-level encoder here. After obtaining the two types of features, the scale conversion from the phoneme level and the character level to the prosody word level is completed by average pooling. Then, the two embeddings are spliced ​​together and sent to the predictor. The text information input of the prosodic phrase level encoder is the same as that of the prosodic word level, but it is consistent with the encoding module at the same time. Referring to the characteristic that the high-level prosodic features in the prosodic boundary structure are built on the basis of the bottom level, the prosodic word level prosodic embedding vector is added as an additional conditional input, where the prosodic embedding vector is sampled from the posterior distribution obtained by encoding the multi-level prosodic encoder. At this time, the training parameter part includes the acoustic model body and the multi-level prosodic encoder. For each sample in the training stage, its Mel spectrum map, phoneme sequence and word-level sequence are sent to the encoder module of the prosodic encoder and synthesizer respectively. Usually, the model obtains the tangent components of the Mel spectrum at the time resolution of the prosodic word and the prosodic phrase respectively under the supervision of the phoneme boundary alignment information and the alignment relationship between the prosodic word, prosodic phrase and phoneme obtained by Montreal-Forced-Aligner (MFA), and then uses the corresponding encoder to obtain the corresponding posterior distribution at these time scales, and then obtains the prosodic embedding vector corresponding to the prosodic word and prosodic phrase by random sampling. Then, using the alignment relationship, the aligned data is reversely expanded and aligned to form the decoder input together with the text encoder output, and then the decoder is used to complete the frame-level decoding of the Mel spectrum. The training standard of the model is similar to that of FastSpeech, including Mel spectrum loss and phoneme time prediction loss. In addition, the kl divergence loss between the VAE training standard and the standard normal distribution is added to train the multi-level prosody encoder.

[0103] In some embodiments, Fig. 9As shown, a flow chart of a speech synthesis method of a speech synthesis model based on prosodic boundary information and VAE structure is provided. The server can establish a speech synthesis data set including prosodic labels, text and phoneme information and corresponding speech data. The server can combine the dictionary and use the speech synthesis data set to train a speech synthesis (acoustic) model based on prosodic boundary information and VAE structure. Specifically, the server can use the speech synthesis data in the speech synthesis data set to model the phoneme information, text information and prosodic information related to the prosodic boundary, and then first train the speech synthesis network and the (multi-level) prosodic coding network. After the speech synthesis network and the (multi-level) prosodic coding network are trained for a certain period of time, as the prosodic coding network kl The convergence of the loss, adding the training of the (multi-level) prosody prediction network, using the results of the prosody encoding network to train the (multi-level) prosody prediction network. During the training of the prosody prediction network, in order to solve the mismatch between training and reasoning caused by the gap between the prosody encoding information predicted in the reasoning stage and the real counterpart, during the training, not all the encoder-encoded features are provided to the decoder. Instead, the prosody representation of the encoder is replaced with the prosody representation of the prosody word level and the prosody phrase level predicted by the prosody prediction network in a certain proportion to reduce the gap between training and reasoning. The server can combine the scheduling sampling mechanism to make the speech synthesis network Under the guidance of the multi-level prosody prediction network, the training converges to obtain a trained speech synthesis model based on prosody boundary information and VAE structure. During the training process of the speech synthesis (acoustic) model based on prosody boundary information and VAE structure, the server can also judge the overall model loss and the prosody listening test of the synthesized speech under the guidance of the prosody encoder. After the prosody encoding has been initially stabilized, the model adds the prosody predictor to the training process, uses the mean and variance of the prediction vector and the mean and variance of the encoding posterior distribution to train the prosody predictor, and stops the gradient flow to ensure that the prosody prediction error does not affect the parameter learning of the prosody encoder. For each training speech, the text sequence and phoneme sequence of the training speech are used to predict the mean and variance of the mixed Gaussian distribution of all prosody words and prosody phrases after the process, and obtain the prosody embedding vector by sampling.When using a speech synthesis (acoustic) model based on prosodic boundary information and a VAE structure for speech generation, the server can input text, prosodic boundary information, and a phoneme sequence into the speech synthesis (acoustic) model based on prosodic boundary information and a VAE structure. The speech synthesis (acoustic) model based on prosodic boundary information and a VAE structure can generate a corresponding mel-gram, and then generate synthesized speech based on the mel-gram. For example, the server segments the text by word and converts it into a numerical sequence corresponding to the text according to a dictionary, segments the phonemes by word and converts them into a numerical sequence corresponding to the phonemes according to a dictionary, processes the prosodic boundary information (prosodic boundary labels) into the matrix format required by the model, and the server inputs linguistic features such as text information, phoneme information, and prosodic boundary information into the speech synthesis model based on prosodic boundary information and a VAE structure. The speech synthesis model based on prosodic boundary information and a VAE structure can output a reconstructed speech mel-gram, and then generate a mel-gram of speech corresponding to the text, and finally generate speech based on the mel-gram.

[0104] In this embodiment, based on the speech synthesis network, a variational autoencoder and a priori information modeling of prosodic boundaries in the text are combined to generate multi-level prosodic features that conform to the prosodic hierarchy of Chinese, and word-level semantic information other than the phoneme sequence is introduced by introducing a pre-trained BERT encoder. This can improve the model's ability to predict speech prosody, thereby generating synthetic speech with accurate prosody and improving the accuracy of synthetic speech.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a speech synthesis model generation device based on prosodic boundary information and VAE structure for implementing the above-mentioned speech synthesis model generation method based on prosodic boundary information and VAE structure. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more speech synthesis model generation devices based on prosodic boundary information and VAE structure provided below can refer to the limitations of the speech synthesis model generation method based on prosodic boundary information and VAE structure above, and will not be repeated here.

[0107] In an exemplary embodiment, Fig.10 As shown, a speech synthesis model generation device based on prosodic boundary information and VAE structure is provided, including: a training data acquisition module 1002, a synthesis network training module 1004, an encoding network training module 1006, a prediction network training module 1008 and a model generation module 1010, wherein:

[0108] The training data acquisition module 1002 is used to acquire a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech.

[0109] The synthesis network training module 1004 is used to train the speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network.

[0110] The coding network training module 1006 is used to train a prosodic coding network according to the sample speech and the prosodic boundary information corresponding to the sample speech to obtain a trained prosodic coding network.

[0111] The prediction network training module 1008 is used to train a prosody prediction network according to the sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech to obtain a trained prosody prediction network.

[0112] The model generation module 1010 is used to generate the speech synthesis model based on the prosodic boundary information and the VAE structure according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

[0113] In one of the exemplary embodiments, the prediction network training module 1008 is further specifically used to input the phoneme information corresponding to the sample speech into the trained speech synthesis network to obtain the phoneme feature information corresponding to the phoneme information; input the phoneme feature information and the text information corresponding to the sample speech into the prosody prediction network to obtain a prosody prediction result; input the sample speech and the prosody boundary information corresponding to the sample speech into the trained prosody encoding network to obtain prosody feature information; and train the prosody prediction network based on the prosody prediction result and the prosody feature information.

[0114] In one of the exemplary embodiments, the prediction network training module 1008 is further used to select candidate prosody feature information from the prosody feature information; and train the prosody prediction network according to the prosody prediction result and the candidate prosody feature information. In one of the exemplary embodiments, the synthesis network training module 1004 is further used to input the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network to obtain the latent variable feature information corresponding to the sample speech; generate the speech synthesis result corresponding to the sample speech according to the latent variable feature information through the speech synthesis network; and train the speech synthesis network according to the speech synthesis result and the sample speech.

[0115] In one of the exemplary embodiments, the above-mentioned encoding network training module 1006 is specifically used to input the phoneme information corresponding to the sample speech into the speech synthesis network to obtain the speech synthesis result corresponding to the sample speech; input the speech synthesis result into the prosody encoding network to obtain the prosody feature information corresponding to the speech synthesis result; and train the prosody encoding network according to the prosody feature information corresponding to the speech synthesis result and the prosody boundary information corresponding to the sample speech.

[0116] Each module in the speech synthesis model generation device based on prosody boundary information and VAE structure can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0117] In an exemplary embodiment, a speech generation device based on a speech synthesis model of prosody boundary information and a VAE structure is provided, comprising: an input module and a generation module, wherein:

[0118] The input module is used to input the text information corresponding to the speech to be generated into the prosody prediction network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure to obtain the prosody prediction result corresponding to the speech to be generated.

[0119] A generation module is used to input the phoneme information corresponding to the speech to be generated and the prosody prediction result corresponding to the speech to be generated into the speech synthesis network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure to generate the speech to be generated.

[0120] Each module in the speech generation device of the speech synthesis model based on prosody boundary information and VAE structure can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0121] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating a speech synthesis model based on rhythmic boundary information and VAE structure is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0122] Those skilled in the art will understand that Fig.11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0127] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a speech synthesis model based on prosodic boundary information and VAE structure, characterized in that: The method comprises: Acquire a training data set; the training data set includes sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech; Training a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech to obtain a trained speech synthesis network; Training a prosody coding network according to the sample speech and prosody boundary information corresponding to the sample speech to obtain a trained prosody coding network; Training a prosody prediction network according to the sample speech and text information, phoneme information, and prosody boundary information corresponding to the sample speech to obtain a trained prosody prediction network; The speech synthesis model based on prosodic boundary information and VAE structure is generated according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

2. The method according to claim 1, characterized in that The step of training a prosody prediction network according to the sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech comprises: Inputting the phoneme information corresponding to the sample speech into the trained speech synthesis network to obtain the phoneme feature information corresponding to the phoneme information; Inputting the phoneme feature information and the text information corresponding to the sample speech into the prosody prediction network to obtain a prosody prediction result; Inputting the sample speech and the prosodic boundary information corresponding to the sample speech into the trained prosodic coding network to obtain prosodic feature information; The rhythm prediction network is trained according to the rhythm prediction result and the rhythm feature information.

3. The method according to claim 2, characterized in that The step of training the prosody prediction network according to the prosody prediction result and the prosody feature information comprises: Filtering candidate prosody feature information from the prosody feature information; The prosody prediction network is trained according to the prosody prediction result and the candidate prosody feature information.

4. The method according to claim 1, characterized in that: The step of training a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech comprises: Inputting the sample speech and the phoneme information corresponding to the sample speech into the speech synthesis network to obtain latent variable feature information corresponding to the sample speech; Generate a speech synthesis result corresponding to the sample speech according to the latent variable feature information through the speech synthesis network; The speech synthesis network is trained according to the speech synthesis result and the sample speech.

5. The method according to claim 1, characterized in that The step of training a prosody coding network according to the sample speech and prosody boundary information corresponding to the sample speech comprises: Inputting the phoneme information corresponding to the sample speech into the speech synthesis network to obtain a speech synthesis result corresponding to the sample speech; Inputting the speech synthesis result into the prosody coding network to obtain prosody feature information corresponding to the speech synthesis result; The prosodic coding network is trained according to the prosodic feature information corresponding to the speech synthesis result and the prosodic boundary information corresponding to the sample speech.

6. A speech generation method based on a speech synthesis model based on prosodic boundary information and a VAE structure, characterized in that: The method comprises: Inputting text information corresponding to the speech to be generated into a prosody prediction network in a pre-trained speech synthesis model based on prosody boundary information and VAE structure to obtain a prosody prediction result corresponding to the speech to be generated; The phoneme information corresponding to the speech to be generated and the prosody prediction result corresponding to the speech to be generated are input into the speech synthesis network in the pre-trained speech synthesis model based on prosody boundary information and VAE structure to generate the speech to be generated.

7. A speech synthesis model generation device based on prosodic boundary information and VAE structure, characterized in that: The device comprises: A training data acquisition module, used to acquire a training data set; the training data set includes a sample speech and prosodic boundary information, text information and phoneme information corresponding to the sample speech; A synthesis network training module, used to train a speech synthesis network according to the sample speech and the phoneme information corresponding to the sample speech, to obtain a trained speech synthesis network; A coding network training module, used for training a prosodic coding network according to the sample speech and the prosodic boundary information corresponding to the sample speech, to obtain a trained prosodic coding network; A prediction network training module, used for training a prosody prediction network according to the sample speech and text information, phoneme information and prosody boundary information corresponding to the sample speech, to obtain a trained prosody prediction network; A model generation module is used to generate the speech synthesis model based on prosodic boundary information and VAE structure according to the trained speech synthesis network, the trained prosodic encoding network and the trained prosodic prediction network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images