Speech synthesis methods, prediction network training methods, servers and storage media

Abstract

This application relates to a speech synthesis method, a prediction network training method, a server, and a storage medium. The speech synthesis method includes: acquiring timbre feature data about a target user and initial audio data about the audio to be synthesized; inputting the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and inputting the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain predicted duration data for each phoneme segment output by the phoneme duration prediction network; and performing feature fusion between the timbre feature data of the target user and the predicted fundamental frequency data and predicted duration data of each phoneme segment to generate synthesized speech audio for the target user. This method can improve the efficiency of synthesized speech audio and ensure high quality of the synthesized speech audio.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a speech synthesis method, a training method for a prediction network, a server, and a storage medium. Background Technology

[0002] With the development of internet technology, speech synthesis, as a new application technology field, utilizes audio and word synthesis techniques to create synthesized speech audio tailored to specific users via computers. Therefore, speech synthesis technology has considerable application value and promising prospects in fields such as virtual singers, record production, and digital music creation.

[0003] In traditional high-quality speech synthesis methods, the process typically involves first manually listening to multiple audio segments of the target user's speech, then using these audio segments to train a speech synthesis model for the target user, and finally using the speech synthesis model to synthesize the target user's speech audio.

[0004] However, current speech synthesis methods targeting specific users require a large amount of audio recordings of the target user's speech to be manually listened to as training data in order to train a high-quality speech synthesis model. Therefore, when the amount of audio recordings of the target user's speech is insufficient, the quality of the speech audio synthesized by the speech synthesis model targeting specific users is not high enough. Summary of the Invention

[0005] Therefore, it is necessary to provide a speech synthesis method, a prediction network training method, a server, and a storage medium that can improve the quality of synthesized speech audio, in order to address the above-mentioned technical problems.

[0006] According to a first aspect of the present disclosure, a speech synthesis method is provided, comprising:

[0007] Acquire timbre feature data about the target user and initial audio data about the audio to be synthesized; the initial audio data is speech audio generated by the target user's associated user in response to the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments;

[0008] The initial audio data of the audio to be synthesized is input into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and

[0009] The initial audio data of the audio to be synthesized is input into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0010] The timbre feature data of the target user is fused with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data, and the synthetic feature data is used to generate synthetic speech audio for the target user.

[0011] In an exemplary embodiment, the step of fusing the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data includes:

[0012] The initial audio data is segmented into phonemes to obtain multiple corresponding phoneme segments;

[0013] Each of the phoneme segments is input into a pre-trained feature encoding network to obtain phoneme feature data for each of the phoneme segments output by the feature encoding network.

[0014] According to the order between the phoneme segments, the phoneme feature data, predicted fundamental frequency data and predicted duration data corresponding to each phoneme segment are sequentially fused with the first feature to obtain the fused feature data corresponding to each phoneme segment.

[0015] The timbre feature data of the target user is fused with the fusion feature data of each phoneme segment to generate the synthesis feature data for the audio to be synthesized.

[0016] In one exemplary embodiment, after obtaining the fusion feature data corresponding to each of the phoneme segments, the method further includes:

[0017] The fusion feature data of each phoneme segment is segmented into single frames according to the prediction duration of the corresponding phoneme segment to obtain multiple single-frame fusion feature data corresponding to each phoneme segment.

[0018] The step of performing a second feature fusion between the timbre feature data of the target user and the fusion feature data of each phoneme segment to generate synthetic feature data for the audio to be synthesized includes:

[0019] According to the order of the single-frame fusion feature data, the single-frame fusion feature data is sequentially fused with the timbre feature data of the target user to generate the synthesized feature data for the audio to be synthesized.

[0020] In an exemplary embodiment, the synthetic feature data is used to characterize the Mel spectrum features of the audio to be synthesized, and the synthesized speech audio of the target user is a speech waveform file of a corresponding media format;

[0021] The generation of synthesized speech audio for the target user includes:

[0022] The Mel spectrum features are input into a pre-trained feature decoding network to obtain Mel spectrum data for the audio to be synthesized, which is output by the feature decoding network.

[0023] The Mel spectrum data of the audio to be synthesized is converted into a waveform to obtain a speech waveform file corresponding to the media format of the synthesized speech audio.

[0024] In one exemplary embodiment, obtaining vocal timbre characteristic data about a target user includes:

[0025] Acquire initial audio data for the target user and a text sequence for the audio to be synthesized; the text sequence is used to express the audio content of the audio to be synthesized.

[0026] The initial audio data of the target user is subjected to timbre extraction to obtain timbre feature data specific to the target user.

[0027] In one exemplary embodiment, obtaining initial audio data about the audio to be synthesized includes:

[0028] From a preset audio database, an audio data set that matches the text sequence of the audio to be synthesized is determined; the audio data set includes multiple audio data, and each audio data is a voice audio of an associated user corresponding to the target user expressing the text sequence;

[0029] In the audio data set, the target audio data with the highest similarity to the timbre feature data of the target user is identified, and the target audio data is used as the initial audio data for the audio to be synthesized.

[0030] In an exemplary embodiment, the step of inputting the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and inputting the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network, includes:

[0031] From the initial audio data of the audio to be synthesized, obtain the timbre feature data and phoneme feature data for each phoneme segment;

[0032] The timbre feature data and phoneme feature data of each phoneme segment are input into the fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and

[0033] The timbre feature data and phoneme feature data of each phoneme segment are input into the phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0034] According to a second aspect of the present disclosure, a method for training a prediction network is provided, comprising:

[0035] Acquire multiple training audio samples for the training object; each training audio sample is based on multiple phoneme segments.

[0036] Phoneme extraction and timbre extraction are performed on multiple training audio samples respectively to obtain feature data for each training audio sample; the feature data includes phoneme feature data and timbre feature data.

[0037] The feature data of each training audio sample is input into the feature prediction network to be trained to obtain the predicted feature data for each training audio sample output by the feature prediction network to be trained; the predicted feature data includes the predicted fundamental frequency data or the predicted duration data for each phoneme segment in the training audio sample.

[0038] Adjust the network parameters of the feature prediction network to be trained according to the predicted feature data to obtain the pre-trained feature prediction network.

[0039] The pre-trained feature prediction network includes a fundamental frequency prediction network or a phoneme duration prediction network. The fundamental frequency prediction network is used to predict the fundamental frequency of the initial audio data of the audio to be synthesized, and outputs the predicted fundamental frequency data for each phoneme segment in the initial audio data. The phoneme duration prediction network is used to predict the duration of the initial audio data of the audio to be synthesized, and outputs the predicted duration data for each phoneme segment in the initial audio data.

[0040] In one exemplary embodiment, after obtaining the pre-trained feature prediction network, the method further includes:

[0041] Obtain audio data about the target object;

[0042] Phoneme extraction or timbre extraction is performed on the audio data of the target object to obtain target phoneme feature data or target timbre feature data;

[0043] The network parameters of the fundamental frequency prediction network are adjusted according to the target phoneme feature data to obtain the corresponding adjusted fundamental frequency prediction network; or, the network parameters of the phoneme duration prediction network are adjusted according to the target timbre feature data to obtain the corresponding adjusted phoneme duration prediction network.

[0044] According to a third aspect of the present disclosure, a speech synthesis apparatus is provided, comprising:

[0045] The data acquisition unit is configured to acquire timbre feature data about the target user and initial audio data about the audio to be synthesized; the initial audio data is speech audio generated by the target user's associated user in response to the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments;

[0046] The first prediction unit is configured to input the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network.

[0047] The second prediction unit is configured to input the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0048] The speech synthesis unit is configured to perform feature fusion of the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthesized feature data, and to use the synthesized feature data to generate synthesized speech audio for the target user.

[0049] According to a fourth aspect of the present disclosure, a training apparatus for a prediction network is provided, comprising:

[0050] The sample acquisition unit is configured to acquire multiple training audio samples for a training object; each training audio sample is based on multiple phoneme segments.

[0051] The feature extraction unit is configured to perform phoneme extraction and timbre extraction on a plurality of training audio samples respectively, to obtain feature data for each training audio sample; the feature data includes phoneme feature data and timbre feature data.

[0052] The network prediction unit is configured to sequentially input the feature data of each of the training audio samples into the feature prediction network to be trained, and obtain the predicted feature data output by the feature prediction network to be trained for each of the training audio samples; the predicted feature data includes the predicted fundamental frequency data or the predicted duration data for each of the phoneme segments in the training audio samples.

[0053] The network training unit is configured to perform adjustments to the network parameters of the feature prediction network to be trained based on each of the predicted feature data, so as to obtain a pre-trained feature prediction network.

[0054] The pre-trained feature prediction network includes a fundamental frequency prediction network or a phoneme duration prediction network. The fundamental frequency prediction network is used to predict the fundamental frequency of the initial audio data of the audio to be synthesized, and outputs the predicted fundamental frequency data for each phoneme segment in the initial audio data. The phoneme duration prediction network is used to predict the duration of the initial audio data of the audio to be synthesized, and outputs the predicted duration data for each phoneme segment in the initial audio data.

[0055] According to a fifth aspect of the present disclosure, a server is provided, comprising:

[0056] processor;

[0057] Memory for storing the executable instructions of the processor;

[0058] The processor is configured to execute the executable instructions to implement the speech synthesis method as described in any of the preceding claims, and / or the training method for the prediction network.

[0059] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium including a computer program that, when executed by a processor of a server, enables the server to perform the speech synthesis method as described in any of the preceding claims, and / or the training method for a prediction network.

[0060] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including program instructions that, when executed by a processor of a server, enable the server to perform the speech synthesis method as described in any of the preceding claims, and / or the training method for a prediction network.

[0061] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0062] The method first acquires timbre feature data about the target user and initial audio data about the audio to be synthesized. The initial audio data consists of speech audio generated by associated users of the target user for the audio content to be synthesized, and is composed of multiple phoneme segments. Then, the initial audio data of the audio to be synthesized is input into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network. The initial audio data of the audio to be synthesized is also input into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network. Finally, the timbre feature data of the target user is fused with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data, and the synthetic speech audio for the target user is generated using the synthetic feature data. In contrast to existing technologies that train a speech synthesis model for a target user using a large amount of audio data, this application can generate synthesized speech audio for a target user using only the timbre features of the target user, the predicted fundamental frequency features of users associated with the target user, and the predicted phoneme duration features, even with a limited amount of audio data. This optimizes the speech audio production process and reduces manpower and time costs. Furthermore, by generating synthesized speech audio for the target user using timbre feature data and initial audio data of the audio to be synthesized, the efficiency of speech audio synthesis is improved and the quality of the synthesized speech audio is guaranteed, thereby enhancing the user experience.

[0063] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0065] Figure 1 This is an application environment diagram illustrating a speech synthesis method according to an exemplary embodiment.

[0066] Figure 2 This is a flowchart illustrating a speech synthesis method according to an exemplary embodiment.

[0067] Figure 3 This is a flowchart illustrating a step of determining timbre feature data for a target user according to an exemplary embodiment.

[0068] Figure 4This is a flowchart illustrating a step of determining initial audio data for audio to be synthesized, according to an exemplary embodiment.

[0069] Figure 5 This is a flowchart illustrating a step for obtaining predicted fundamental frequency data and predicted duration data according to an exemplary embodiment.

[0070] Figure 6 This is a flowchart illustrating a step of generating synthesized speech audio according to an exemplary embodiment.

[0071] Figure 7 This is a schematic diagram of a module for generating synthesized speech audio according to an exemplary embodiment.

[0072] Figure 8 This is a flowchart illustrating the training of a prediction network according to an exemplary embodiment.

[0073] Figure 9 This is a flowchart illustrating a speech synthesis method according to another exemplary embodiment.

[0074] Figure 10 This is a block diagram illustrating a speech synthesis method according to another exemplary embodiment.

[0075] Figure 11 This is a block diagram illustrating a speech synthesis apparatus according to an exemplary embodiment.

[0076] Figure 12 This is a block diagram of a training apparatus for a prediction network according to an exemplary embodiment.

[0077] Figure 13 This is a block diagram illustrating a server for speech synthesis according to an exemplary embodiment.

[0078] Figure 14 This is a block diagram illustrating a computer-readable storage medium for speech synthesis according to an exemplary embodiment.

[0079] Figure 15 This is a block diagram illustrating a computer program product for speech synthesis according to an exemplary embodiment. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0081] The term "and / or" in the embodiments of this application refers to any and all possible combinations including one or more of the associated listed items. It should also be noted that, when used in this specification, "including / comprising" specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or components and / or groups thereof.

[0082] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0083] Furthermore, although the terms "first," "second," etc., are used repeatedly in this application to describe various operations (or various components, or various applications, or various instructions, or various data), these operations (or components, or applications, or instructions, or data) should not be limited by these terms. These terms are only used to distinguish one operation (or component, or application, or instruction, or data) from another operation (or component, or application, or instruction, or data). For example, a first prediction network can be called a second prediction network, and a second prediction network can be called a first prediction network; the only difference is the scope they encompass, but it does not depart from the scope of this application. Both the first prediction network and the second prediction network are collections of various types of network models for predicting audio features, but they are not collections of network models for predicting audio features of the same category.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0085] The method for creating audio templates provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on a web server.

[0086] In some embodiments, reference Figure 1First, server 104 acquires timbre feature data about the target user and initial audio data about the audio to be synthesized. The initial audio data is the speech audio generated by the target user's associated users for the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments. Then, server 104 inputs the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network. Also, it inputs the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network. Finally, server 104 performs feature fusion with the timbre feature data of the target user, the predicted fundamental frequency data of each phoneme segment, and the predicted duration data to obtain synthesis feature data, and uses the synthesis feature data to generate synthesized speech audio for the target user.

[0087] In some embodiments, terminal 102 (such as a mobile terminal or a fixed terminal) can be implemented in various forms. Terminal 102 can be a mobile terminal, including mobile phones, smartphones, laptops, portable handheld devices, personal digital assistants (PDAs), tablet computers (PADs), etc., capable of performing speech synthesis based on the target user's voice characteristics data and initial audio data of the audio to be synthesized. Terminal 102 can also be a fixed terminal, including automated teller machines (ATMs), automated kiosks, digital TVs, desktop computers, fixed-line computers, etc., capable of performing speech synthesis based on the target user's voice characteristics data and initial audio data of the audio to be synthesized.

[0088] Hereinafter, it is assumed that terminal 102 is a fixed terminal. However, those skilled in the art will understand that, if there are operations or elements specifically designed for mobile purposes, the construction according to the embodiments disclosed in this application can also be applied to mobile type terminal 102.

[0089] In some embodiments, the data processing component running on server 104 may load any of the various additional server applications and / or middleware applications being executed, such as HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), CGI (Common Gateway Interface), RDBMS (Relational Database Management System), etc.

[0090] In some embodiments, server 104 may implement the corresponding network functions through deployed servers (such as a standalone server or a server cluster consisting of multiple servers). Server 104 may also be adapted to run one or more application services or software components that provide the terminal 102 described in the foregoing disclosure.

[0091] In some embodiments, the application service may include a service interface that provides users with initial audio data of various audio samples to be synthesized, as well as corresponding program services, etc. The software component may include, for example, an application (SDK) or client (APP) that performs feature prediction and speech synthesis on the audio data selected by the target user.

[0092] In some embodiments, the application or client provided by server 104, which has the function of feature prediction and speech synthesis of audio data based on the audio data selected by the target user, includes a portal port that provides one-to-one application services to the user in the foreground and multiple business systems that perform data processing in the background, so as to extend the application of feature prediction and speech synthesis functions of audio data to the APP or client, so that the user can use and access speech synthesis-related functions anytime and anywhere.

[0093] In some embodiments, the feature prediction and speech synthesis functions for audio data in an APP or client can be a computer program running in user mode to perform one or more specific tasks. This program can interact with the user and has a visual user interface. The APP or client can include two parts: a graphical user interface (GUI) and an engine, which together provide users with a variety of application services in the form of a user interface, creating a digital client system.

[0094] In some embodiments, users can input corresponding code data or control parameters into the APP or client through a preset input device or automatic control program to execute application services of the computer program in the server 104 and display application services in the user interface.

[0095] As an example, when a user needs to synthesize a song for a target user in terminal 102, the user can input timbre feature data of the target user and initial audio data of the audio to be synthesized into terminal 102 through an input device. Then, the prediction network in server 104 predicts the predicted fundamental frequency data and predicted duration data of each phoneme segment in the initial audio data. Server 104 then fuses the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain the synthesized speech audio for the target user. Finally, server 104 sends the synthesized speech audio for the target user to terminal 102 so that the synthesized speech audio can be displayed in the APP or client running on terminal 102.

[0096] In some embodiments, the operating system running the app or client may include various versions of Microsoft... Apple and / or Linux operating system, various commercial or similar Operating systems (including but not limited to various GNU / Linux operating systems, Google) OS and / or mobile operating systems, such as Phone OS OS OS operating systems, as well as other online or offline operating systems, are not specifically limited here.

[0097] In some embodiments, such as Figure 2 As shown, a speech synthesis method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0098] Step S11: Obtain timbre feature data about the target user and initial audio data about the audio to be synthesized.

[0099] In some embodiments, the server obtains timbre feature data about the target user and initial audio data about the audio to be synthesized from a terminal application (such as a mobile phone, tablet, etc.). The timbre feature data of the target user is extracted in advance from any audio data of the target user, and is used to characterize the timbre features of the target user.

[0100] For example, the server first obtains the audio of the target user singing a song A. Then, the server inputs the audio of song A into a preset timbre extraction model to obtain the output timbre feature data of the target user, and stores the timbre feature data in the APP on the mobile terminal.

[0101] In some embodiments, the initial audio data of the audio to be synthesized is speech audio generated by an associated user corresponding to the target user for the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments.

[0102] Among them, the associated user is a user account whose timbre features meet the preset similarity threshold with the timbre features of the target user, and the speech audio generated by the user account for the audio to be synthesized (including recitation, singing, etc.) is the initial audio data of the audio to be synthesized.

[0103] As an example, in a user database including multiple user accounts and the target audio of each user account singing a cover of the audio to be synthesized, the server first extracts multiple timbre features from the target audio of each user account, then performs similarity matching between these multiple timbre features and the timbre features of the target user, and takes the user account with the highest similarity to the corresponding timbre features as the associated user, and finally takes the target audio of the associated user singing a cover of the audio to be synthesized as the initial audio data of the audio to be synthesized.

[0104] In some embodiments, the initial audio data may be a released, officially released version of a music song, or a local song recorded by the terminal application (e.g., a live song recorded offline by the terminal application and a web song recorded online).

[0105] In some embodiments, the server obtains the timbre feature data of the target user and / or the timbre feature data of associated users based on the audio depth features of the corresponding user's speech audio. For example, the server first obtains a segment of song A sung by the target user, then uses the Spleeter algorithm to segment song A, and projects the segmented song A into a low-dimensional space to reduce dimensionality and compress information data to extract audio depth features of song A. Then, the server uses a multilayer perceptron based on MLP to classify the audio depth features to obtain audio depth features of the dry audio. Finally, the compressed low-dimensional features are restored to the original dimensionality of the dry audio, and Mel-spectral data is extracted from the dry audio to represent the timbre feature data of the target user based on the Mel-spectral data.

[0106] Step S12: Input the initial audio data of the audio to be synthesized into the pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network.

[0107] In some embodiments, the pre-trained fundamental frequency prediction network is a convolutional neural network model pre-trained by the server using speech audio from multiple user accounts. This convolutional neural network model can be an acoustic model such as the Fast Speech2 model.

[0108] The fundamental frequency prediction network is used to extract phoneme-level timbre features and phoneme features from the input speech audio data, and to predict the fundamental frequency value of the input speech audio data based on the extracted timbre features and phoneme features, so as to generate phoneme-level predicted fundamental frequency data based on the predicted fundamental frequency value.

[0109] Step S13: Input the initial audio data of the audio to be synthesized into the pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0110] In some embodiments, the pre-trained phoneme duration prediction network is a convolutional neural network model pre-trained by the server using speech audio from multiple user accounts. This convolutional neural network model can be an acoustic model such as the FastSpeech2 model.

[0111] Among them, the phoneme duration prediction network is used to extract phoneme-level timbre features and phoneme features from the input speech audio data, and to predict the phoneme duration of the input speech audio data based on the extracted timbre features and phoneme features, so as to use the predicted phoneme duration to determine the predicted duration data of the speech audio data at the phoneme level.

[0112] Step S14: Perform feature fusion between the target user's timbre feature data and the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data, and use the synthetic feature data to generate synthetic speech audio for the target user.

[0113] In some embodiments, the timbre feature data of the target user is represented based on the timbre feature vector, and the predicted fundamental frequency data of each phoneme segment is represented based on the fundamental frequency feature vector, and the predicted duration data is represented based on the phoneme feature vector, and the vector dimensions of the timbre feature vector, the fundamental frequency feature vector and the phoneme feature vector are the same.

[0114] In some embodiments, the server fuses the timbre feature vector of the target user with the fundamental frequency feature vector and the phoneme feature vector of each corresponding phoneme segment, which have the same vector dimension, to obtain a fused feature vector for the target user. Then, the fused feature vector is subjected to data decoding and data transcoding processes in sequence to generate synthetic speech audio for the target user.

[0115] In the aforementioned speech synthesis process, the server first acquires timbre feature data about the target user and initial audio data about the audio to be synthesized. The initial audio data is the speech audio generated by a user associated with the target user for the audio content to be synthesized, and is composed of multiple phoneme segments. Then, the initial audio data is input into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment. The initial audio data is also input into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment. Finally, the timbre feature data of the target user is fused with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthesis feature data, which is then used to generate synthesized speech audio for the target user. In this way, on the one hand, unlike the existing technology that trains a speech synthesis model for a target user using a large amount of speech audio as training data, this application can generate synthesized speech audio for a target user using only the timbre features of the target user, the predicted fundamental frequency features of users associated with the target user, and the predicted phoneme duration features, even with a small amount of speech audio for the target user. This optimizes the process of synthesizing speech audio and reduces the consumption of manpower and time costs. On the other hand, by generating synthesized speech audio for the target user using timbre feature data of the target user and initial audio data of the audio to be synthesized, the efficiency of synthesizing speech audio is improved and the quality of the synthesized speech audio is guaranteed, thereby enhancing the user experience.

[0116] Those skilled in the art will understand that the methods disclosed in the above-described specific embodiments can be implemented in more specific ways. For example, the implementation described above, in which the server fuses the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data, and uses the synthetic feature data to generate synthetic speech audio for the target user, is merely illustrative.

[0117] For example, the fundamental frequency prediction network in the server predicts the fundamental frequency of the initial audio data of the synthesized audio; or the phoneme duration prediction network in the server predicts the phoneme duration of the initial audio data of the synthesized audio, etc. These are just one set of methods. In actual implementation, there may be other ways of dividing the data. For example, the predicted fundamental frequency data and predicted duration data for phoneme segments can be combined or integrated into another system, or some features can be ignored or not performed.

[0118] In one exemplary embodiment, see Figure 3 , Figure 3This is a flowchart illustrating an embodiment of determining timbre feature data for a target user in this application. In step S11, the process of the server acquiring timbre feature data about the target user can be implemented in the following way:

[0119] Step S111: Obtain initial audio data for the target user and text sequence for the audio to be synthesized.

[0120] In one embodiment, the server first obtains any audio data of a preset duration for the target user, and then performs dry separation on the audio data to obtain initial audio data for the target user.

[0121] For example, the target user first records a 5-minute audio segment. Then, the server obtains the audio segment and performs data segmentation and dry separation on the audio segment to obtain a 3-minute dry audio segment for the target user. This 3-minute dry audio segment is then used as the target user's initial audio data.

[0122] In one embodiment, the audio to be synthesized is the speech audio to be synthesized by the server, and the text sequence is used to express the audio content (i.e., text content) of the speech audio to be synthesized. For example, if the server wants to synthesize a synthesized audio about song A, then song A is the audio to be synthesized, and the lyrics in song A are the text sequence of the audio to be synthesized.

[0123] Step S112: Extract the timbre from the initial audio data of the target user to obtain timbre feature data specific to the target user.

[0124] In some embodiments, the timbre feature data obtained by the server can be obtained based on the audio depth features of the corresponding speech audio.

[0125] For example, the server first acquires a segment of audio from song A, then uses the Spleeter algorithm to segment song A, and projects the segmented song A into a low-dimensional space to reduce dimensionality and compress information data to extract audio depth features related to song A. Next, the server uses a multilayer perceptron based on MLP to classify the audio depth features, obtaining audio depth features related to the dry audio. Then, the compressed low-dimensional features are restored to the original dimensionality of the dry audio, and Mel-spectral data is extracted from the dry audio. Finally, the server uses the Mel-spectral data to represent the timbre characteristics of the target user.

[0126] In one exemplary embodiment, see Figure 4 , Figure 4This is a schematic flowchart illustrating an embodiment of determining the initial audio data for the audio to be synthesized in this application. In step S11, the process of the server obtaining the initial audio data for the audio to be synthesized can be implemented in the following manner:

[0127] Step S113: Determine the set of audio data that matches the text sequence of the audio to be synthesized from the preset audio database.

[0128] In one embodiment, the audio data set includes multiple audio data, and each audio data is a speech audio of an associated user corresponding to a target user expressing a text sequence.

[0129] As an example, if the audio to be synthesized is song A, the server will determine from a pre-defined audio database that there are N user accounts singing audio clips related to song A. Therefore, the set of N segments of song A sung by these N user accounts is the set of audio data that matches the text sequence of the audio to be synthesized.

[0130] Step S114: In the audio data set, identify the target audio data that has the highest similarity to the timbre feature data of the target user, and use the target audio data as the initial audio data for the audio to be synthesized.

[0131] In one embodiment, the server first extracts multiple timbre features from the voice audio corresponding to each user account in the audio data set, and performs similarity matching between these multiple timbre features and the timbre features of the target user to determine the user account with the highest similarity to the corresponding timbre features as the associated user, and uses the voice audio of the associated user regarding the audio to be synthesized as the initial audio data of the audio to be synthesized.

[0132] As an example, the server uses cosine similarity to characterize the similarity between timbre features of speech audio. The cosine similarity formula is as follows:

[0133] Similarity=cos(x)=A*B / ||A||*||B||.

[0134] Here, A and B are the timbre feature vectors of two different user accounts. For example, A is the timbre feature vector of the target user, and B is the timbre feature vector of an associated user in the audio dataset.

[0135] The value represented by Similarity = cos(x) is the similarity between the timbre features of the speech audio. The larger the value, the greater the similarity of the corresponding timbre features; the smaller the value, the smaller the similarity of the corresponding timbre features.

[0136] In one exemplary embodiment, see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of obtaining predicted fundamental frequency data and predicted duration data in this application. In steps S12 and S13, the server obtains the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and the process of inputting the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network can be specifically implemented in the following ways:

[0137] Step a1: Obtain timbre feature data and phoneme feature data for each phoneme segment from the initial audio data of the audio to be synthesized.

[0138] Step a2: Input the timbre feature data and phoneme feature data of each phoneme segment into the fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network.

[0139] Step a3: Input the timbre feature data and phoneme feature data of each phoneme segment into the phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0140] In some embodiments, the server first performs phoneme-level timbre feature extraction and phoneme feature extraction on the initial audio data to be synthesized, to obtain timbre feature data and phoneme feature data for each phoneme segment. Then, the server inputs the extracted timbre feature data and phoneme feature data into a fundamental frequency prediction network to predict the fundamental frequency value, so as to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; or, the server inputs the extracted timbre feature data and phoneme feature data into a phoneme duration prediction network to predict the phoneme duration, so as to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0141] In some embodiments, the fundamental frequency prediction network and / or phoneme duration prediction network are convolutional neural network models pre-trained by the server using speech audio from multiple user accounts. These convolutional neural network models can be acoustic models such as the FastSpeech2 model.

[0142] In one exemplary embodiment, see Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of a process for generating synthesized speech audio according to an embodiment of this application. Figure 7 This is a schematic diagram of a module for generating synthesized speech audio according to an embodiment of this application. In step S14, the process of feature fusion between the server and the target user's timbre feature data, the predicted fundamental frequency data of each phoneme segment, and the predicted duration data can be implemented in the following way:

[0143] Step S141: The initial audio data is segmented into phonemes to obtain multiple corresponding phoneme segments.

[0144] In one embodiment, the server inputs the initial audio data of the audio to be synthesized into a preset timbre recognition network for phoneme segmentation, or a music theorist performs manual phoneme segmentation on the initial audio data of the audio to be synthesized to obtain corresponding multiple phoneme segments.

[0145] Step S142: Input each phoneme segment into the pre-trained feature encoding network to obtain the phoneme feature data for each phoneme segment output by the feature encoding network.

[0146] In one embodiment, the server inputs each phoneme segment into a pre-trained feature encoding network (such as an encoder) for text encoding to obtain phoneme feature data for each phoneme segment. The phoneme feature data is used to characterize the phoneme features of the text content in the initial audio data.

[0147] Step S143: According to the order between each phoneme segment, the phoneme feature data, predicted fundamental frequency data and predicted duration data corresponding to each phoneme segment are sequentially fused with the first feature to obtain the fused feature data corresponding to each phoneme segment.

[0148] In one embodiment, after obtaining the fusion feature data corresponding to each phoneme segment, the server further includes: segmenting the fusion feature data of each phoneme segment into single frames according to the prediction duration of the corresponding phoneme segment, thereby obtaining multiple single-frame fusion feature data corresponding to each phoneme segment. The single-frame fusion feature data is used to characterize the phoneme features, prediction duration features, and prediction fundamental frequency features of the corresponding audio frame in the initial audio data.

[0149] Step S144: Perform a second feature fusion on the timbre feature data of the target user and the fusion feature data of each phoneme segment to generate synthesis feature data for the audio to be synthesized.

[0150] Specifically, the server sequentially performs second feature fusion with the target user's timbre feature data, according to the order of the single-frame fusion feature data, to generate synthetic feature data for the audio to be synthesized.

[0151] Among them, the synthesized feature data is used to express the timbre features of the target user for the audio to be synthesized, the phoneme features of the initial audio data, the predicted duration features, and the predicted fundamental frequency features.

[0152] Step S145: Input the Mel spectrum features into the pre-trained feature decoding network to obtain the Mel spectrum data for the audio to be synthesized, which is output by the feature decoding network.

[0153] In one embodiment, the synthesis feature data for the audio to be synthesized is used to characterize the Mel spectrum features of the audio to be synthesized.

[0154] In one embodiment, the server inputs the Mel spectrum features (i.e., synthesis feature data) of the audio to be synthesized into a feature decoding network for feature decoding to obtain Mel spectrum data for the audio to be synthesized. The Mel spectrum data is used to represent the Mel spectrogram of the audio to be synthesized.

[0155] Step S146: Perform waveform conversion on the Mel spectrum data of the audio to be synthesized to obtain a speech waveform file corresponding to the media format of the synthesized speech audio.

[0156] In one embodiment, the server inputs the Mel spectrum data (i.e., Mel spectrogram) of the audio to be synthesized into a vocoder for waveform conversion to obtain a speech waveform file for the media format corresponding to the synthesized speech audio.

[0157] The synthesized speech audio can be a multimedia file in a media format such as MP3 or MP4.

[0158] In some embodiments, such as Figure 8 As shown, a training method for a prediction network is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0159] Step S21: Obtain multiple training audio samples for the training object.

[0160] In one embodiment, the training objects are multiple user accounts stored in the user database, and the multiple training audio samples of the training objects are multiple voice audio data corresponding to each user account.

[0161] For example, if there are 100 users stored in the user database, and each user has more than 1,000 voice sentences and corresponding text content, then the server will extract N (N < 100) users from the user database as training objects, and S (S < 1,000) voice sentences corresponding to the N users and their corresponding text content as training audio samples.

[0162] In one embodiment, each training audio sample is composed of multiple phoneme segments, and each phoneme segment of each training audio sample can be extracted from the text content corresponding to the speech statement.

[0163] Step S22: Phoneme extraction and timbre extraction are performed on multiple training audio samples to obtain feature data for each training audio sample.

[0164] In one embodiment, the feature data includes phoneme feature data corresponding to phoneme extraction and timbre feature data corresponding to timbre extraction.

[0165] In some embodiments, the server inputs each speech sentence and its corresponding text content into a feature extraction network for feature extraction to obtain phoneme feature data for each speech sentence and timbre feature data corresponding to the timbre extraction.

[0166] Step S23: Input the feature data of each training audio sample into the feature prediction network to be trained, and obtain the predicted feature data of each training audio sample output by the feature prediction network to be trained.

[0167] In one embodiment, the predicted feature data includes predicted fundamental frequency data or predicted duration data for each phoneme segment in the training audio samples.

[0168] In some embodiments, the server inputs the phoneme feature data of each speech sentence and the timbre feature data corresponding to the timbre extraction into a fundamental frequency prediction network to be trained for network training, so as to obtain the predicted fundamental frequency data for the speech sentence output by the fundamental frequency prediction network to be trained. Alternatively, the server inputs the phoneme feature data of each speech sentence and the timbre feature data corresponding to the timbre extraction into a phoneme duration prediction network to be trained for network training, so as to obtain the predicted duration data for the speech sentence output by the phoneme duration prediction network to be trained.

[0169] Step S24: Adjust the network parameters of the feature prediction network to be trained according to each predicted feature data to obtain the pre-trained feature prediction network.

[0170] In one embodiment, the pre-trained feature prediction network includes a fundamental frequency prediction network or a phoneme duration prediction network.

[0171] The fundamental frequency prediction network is used to predict the fundamental frequency of the initial audio data to be synthesized, and outputs the predicted fundamental frequency data for each phoneme segment in the initial audio data. The phoneme duration prediction network is used to predict the duration of the initial audio data to be synthesized, and outputs the predicted duration data for each phoneme segment in the initial audio data.

[0172] In the disclosed embodiments of steps S21-S24 of this application, multiple different training objects and their corresponding multiple training audio samples are used to train the prediction ability of the feature prediction network (including the fundamental frequency prediction network and the phoneme duration prediction network, the encoder and the decoder).

[0173] In some embodiments, the server adjusts the trained feature prediction network based on the actual extracted Mel spectrum data corresponding to each training audio sample and the fundamental frequency value of each frame.

[0174] In some embodiments, the model structure (predictor parameters) of the trained feature prediction network can be two convolutional layers with layer norm followed by a linear layer; the trained encoder and decoder can both be 4-layer FFT blocks.

[0175] In one exemplary embodiment, after step S24, the server may further be implemented in the following manner:

[0176] Step 1: Obtain audio data about the target object.

[0177] For example, the target user first records a 5-minute audio segment. Then, the server obtains the audio segment and performs data segmentation and dry separation on the audio segment to obtain a 3-minute dry audio segment for the target user. This 3-minute dry audio segment is then used as the audio data for the target user that needs to be obtained.

[0178] Step two: Extract phonemes or timbre from the audio data of the target object to obtain target phoneme feature data or target timbre feature data.

[0179] Step 3: Adjust the network parameters of the fundamental frequency prediction network based on the target phoneme feature data to obtain the corresponding adjusted fundamental frequency prediction network. Alternatively...

[0180] Step 4: Adjust the network parameters of the phoneme duration prediction network according to the target timbre feature data to obtain the corresponding adjusted phoneme duration prediction network.

[0181] In the disclosed embodiments of steps one through four of this application, the server uses the phoneme features extracted from the target object to adjust the network parameters of the trained feature prediction network (including the fundamental frequency prediction network, the phoneme duration prediction network, the encoder, and the decoder) in order to make the prediction results or data fusion results of the trained feature prediction network more closely match the speech and audio of the target user.

[0182] To more clearly illustrate the speech synthesis method provided in the embodiments of this disclosure, the following describes the method for creating the audio template using a specific embodiment. In an exemplary embodiment, reference is made to... Figure 9 and Figure 10 , Figure 9 This is a flowchart illustrating a speech synthesis method according to another exemplary embodiment. Figure 10 This is a block diagram illustrating a speech synthesis method according to another exemplary embodiment. The method for creating the audio template is used in server 104 and specifically includes the following:

[0183] Step S31: Obtain the initial audio data of the target user and extract the timbre feature vector from the initial audio data of the target user.

[0184] The initial audio data for the target user consists of any pre-defined segment of speech audio data about the target user, such as 3 minutes, 10 minutes, etc. The initial audio data is then input into a feature extraction model to obtain a timbre feature vector of a pre-defined dimension. This timbre feature vector is an encoded vector representing the timbre characteristics of each audio frame within the audio data segment.

[0185] Step S32: Obtain the initial audio data of the associated user for the audio to be synthesized.

[0186] The system stores multiple user accounts in a pre-defined user database. Among these user accounts, the user account whose timbre characteristics are most similar to those of the target user is considered the associated user.

[0187] The initial audio data for the audio to be synthesized is a segment of audio data containing preset text content about the audio to be synthesized. The preset text content is the text content of the speech to be synthesized, and each character in the preset text content is composed of multiple corresponding phonemes.

[0188] For example, if a text about song A is input into the server, the server will retrieve audio data about song A from multiple user accounts in the preset user database. Then, the server will match the audio data of these multiple user accounts and identify the user account whose timbre characteristics are most similar to those of the target user as the associated user. The audio data of the associated user about song A will then be used as the initial audio data for song A.

[0189] Step S33: Extract phonemes and timbre from the initial audio data to be synthesized to obtain the phoneme sequence and timbre feature vector for the initial audio data.

[0190] The phoneme sequence consists of the phonemes contained in each character in the preset text content.

[0191] Step S34: Input the phoneme sequence into the encoder for text encoding to obtain the phoneme feature vector for each phoneme in the phoneme sequence.

[0192] Step S35: Input the phoneme feature vector of each phoneme and the timbre feature vector of the initial audio data into the fundamental frequency predictor to perform fundamental frequency prediction, and obtain the predicted fundamental frequency value for each phoneme in the phoneme sequence; and input the phoneme feature vector of each phoneme and the timbre feature vector of the initial audio data into the phoneme duration predictor to perform phoneme duration prediction, and obtain the predicted phoneme duration for each phoneme in the phoneme sequence.

[0193] Among them, the prediction duration of a phoneme is used to characterize the time domain occupied by a phoneme in speech audio.

[0194] Step S36: The phoneme feature vectors of each phoneme are sequentially fused with the corresponding predicted fundamental frequency value and predicted duration to obtain a fused feature vector for each phoneme.

[0195] The fusion feature vector of each phoneme is used to characterize the phoneme features and fundamental frequency features expressed within the time range corresponding to the initial audio data.

[0196] Step S37: The fusion feature vector of each phoneme is segmented into single frames according to the time range occupied by the corresponding phoneme in the initial audio data to obtain the fusion feature vector for each audio frame.

[0197] The fused feature vector of each audio frame is used to characterize the phoneme features and fundamental frequency features of the corresponding audio frame in the initial audio data.

[0198] Step S38: Perform feature fusion on the fused feature vector of each audio frame and the timbre feature vector of the target user to obtain a synthesized feature vector for the target user.

[0199] Among them, the synthesized feature vector is used to characterize the timbre features, phoneme features, and fundamental frequency features of the target user for the audio to be synthesized.

[0200] The synthetic feature vector of the target user is based on Mel spectrum feature representation.

[0201] Step S39: Input the synthesized feature vector of the target user into the decoder for audio decoding to obtain synthesized audio data for the target user.

[0202] The synthesized audio data for the target user is based on Mel spectrogram representation.

[0203] Step S40 involves inputting the synthesized audio data for the target user into a vocoder for waveform conversion to obtain a speech waveform file corresponding to the synthesized audio data.

[0204] The audio waveform file for synthesized audio data can be a multimedia file such as MP3 or MP4.

[0205] In contrast to existing technologies that train a speech synthesis model for a target user using a large amount of audio data, this application can generate synthesized speech audio for a target user using only the timbre features of the target user, the predicted fundamental frequency features of users associated with the target user, and the predicted phoneme duration features, even with a limited amount of audio data. This optimizes the speech audio production process and reduces manpower and time costs. Furthermore, by generating synthesized speech audio for the target user using timbre feature data and initial audio data of the audio to be synthesized, the efficiency of speech audio synthesis is improved and the quality of the synthesized speech audio is guaranteed, thereby enhancing the user experience.

[0206] It should be understood that, although Figures 2-10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-10 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0207] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0208] Figure 11 This is a block diagram of a speech synthesis device provided in an embodiment of this application. (Refer to...) Figure 11 The speech synthesis device 10 includes: a data acquisition unit 11, a first prediction unit 12, a second prediction unit 13, and a speech synthesis unit 14.

[0209] The data acquisition unit 11 is configured to acquire timbre feature data about the target user and initial audio data about the audio to be synthesized; the initial audio data about the audio to be synthesized is speech audio generated by an associated user corresponding to the target user for the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments.

[0210] The first prediction unit 12 is configured to input the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network.

[0211] The second prediction unit 13 is configured to input the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

[0212] The speech synthesis unit 14 is configured to perform feature fusion of the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthesized feature data, and to use the synthesized feature data to generate synthesized speech audio for the target user.

[0213] Figure 12 This is a block diagram of a training device for a prediction network provided in an embodiment of this application. (Refer to...) Figure 12 The training device 10A for the prediction network includes: a sample acquisition unit 11A, a feature extraction unit 12A, a network prediction unit 13A, and a network training unit 14A.

[0214] The sample acquisition unit 11A is configured to acquire multiple training audio samples about the training object; each training audio sample is composed of multiple phoneme segments.

[0215] The feature extraction unit 12A is configured to perform phoneme extraction and timbre extraction on multiple training audio samples respectively to obtain feature data for each training audio sample; the feature data includes phoneme feature data and timbre feature data.

[0216] The network prediction unit 13A is configured to input the feature data of each training audio sample into the feature prediction network to be trained, and obtain the predicted feature data for each training audio sample output by the feature prediction network to be trained; the predicted feature data includes the predicted fundamental frequency data or the predicted duration data for each phoneme segment in the training audio sample.

[0217] The network training unit 14A is configured to adjust the network parameters of the feature prediction network to be trained according to the predicted feature data, so as to obtain a pre-trained feature prediction network.

[0218] The pre-trained feature prediction network includes a fundamental frequency prediction network or a phoneme duration prediction network. The fundamental frequency prediction network is used to predict the fundamental frequency of the initial audio data of the audio to be synthesized, and outputs the predicted fundamental frequency data for each phoneme segment in the initial audio data. The phoneme duration prediction network is used to predict the duration of the initial audio data of the audio to be synthesized, and outputs the predicted duration data for each phoneme segment in the initial audio data.

[0219] Figure 13 This is a block diagram of a server 20 provided in an embodiment of this application. For example, server 20 can be an electronic device, an electronic component, or a server array, etc. (Refer to...) Figure 13 Server 20 includes processor 21, which may be a processor set, including one or more processors. Server 20 also includes memory resources represented by memory 22, on which computer programs, such as application programs, are stored. The computer programs stored in memory 22 may include one or more modules, each corresponding to a set of executable instructions. Furthermore, processor 21 is configured to implement, when executing the computer programs, the speech synthesis method and / or the training method for the prediction network as described above.

[0220] In some embodiments, server 20 is an electronic device whose computing system can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 20 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, super servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from companies such as IBM.

[0221] In some embodiments, processor 21 typically controls the overall operation of server 20, such as operations associated with display, data processing, data communication, and recording operations. Processor 21 may include one or more processor components to execute computer programs to perform all or part of the steps of the methods described above. Furthermore, processor components may include one or more modules to facilitate interaction between processor components and other components. For example, processor components may include a multimedia module to facilitate control of the interaction between user server 20 and processor 21 using multimedia components.

[0222] In some embodiments, the processor component in processor 21 may also be referred to as a CPU (Central Processing Unit). The processor component may be an electronic chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor component. Furthermore, the processor component may be implemented using integrated circuit chips.

[0223] In some embodiments, memory 22 is configured to store various types of data to support operation on server 20. Examples of such data include instructions for any application or method operating on server 20, acquired data, messages, images, videos, etc. Memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0224] In some embodiments, the memory 22 can be a memory module, TF card, etc., and can store all information in the server 20, including the input raw data, computer programs, intermediate running results, and final running results. In some embodiments, it stores and retrieves information according to the location specified by the processor. In some embodiments, the server 20 has a memory function and can ensure normal operation because of the memory 22. In some embodiments, the memory 22 of the server 20 can be classified into main memory (RAM) and auxiliary memory (external memory) according to its purpose, or it can be classified into external memory and internal memory. External memory is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage component on the motherboard, which is used to store the currently executing data and programs, but it is only used to temporarily store programs and data. The data will be lost when the power is turned off or the power is cut off.

[0225] In some embodiments, server 20 may further include: a power supply component 23 configured to perform power management of server 20, a wired or wireless network interface 24 configured to connect server 20 to a network, and an input / output (I / O) interface 25. Server 20 may operate on an operating system stored in memory 22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0226] In some embodiments, power supply component 23 provides power to various components of server 20. Power supply component 23 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to server 20.

[0227] In some embodiments, the wired or wireless network interface 24 is configured to facilitate wired or wireless communication between the server 20 and other devices. The server 20 may access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof.

[0228] In some embodiments, the wired or wireless network interface 24 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the wired or wireless network interface 24 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0229] In some embodiments, the input / output (I / O) interface 25 provides an interface between the processor 21 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0230] Figure 14 This is a block diagram of a computer-readable storage medium 30 provided in an embodiment of this application. The computer-readable storage medium 30 stores a computer program 31, which, when executed by a processor, implements the speech synthesis method and / or the prediction network training method as described above.

[0231] If the integrated units of the various functional units in the various embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium 30. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer-readable storage medium 30 includes a computer program 31, which includes several instructions to cause a computer device (which may be a personal computer, system server, or network device, etc.), an electronic device (e.g., MP3, MP4, etc., or a mobile phone, tablet computer, wearable device, etc., or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0232] Figure 15 This is a block diagram of a computer program product 40 provided in an embodiment of this application. The computer program product 40 includes program instructions 41, which can be executed by the processor of server 20 to implement the speech synthesis method and / or the prediction network training method as described above.

[0233] Those skilled in the art will understand that embodiments of this application may provide a speech synthesis method, a prediction network training method, a speech synthesis apparatus 10, a prediction network training apparatus 10A, a server 20, a computer-readable storage medium 30, or a computer program product 40. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product 40 embodied on one or more computer program instructions 41 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] This application is described with reference to flowchart illustrations and / or block diagrams of a speech synthesis method, a prediction network training method, a speech synthesis apparatus 10, a prediction network training apparatus 10A, a server 20, a computer-readable storage medium 30, or a computer program product 40 according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by the computer program product 40. These computer program products 40 can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that program instructions 41, executable by the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0235] These computer program products 40 may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing device to function in a particular manner, such that program instructions 41 stored in the computer program product 40 produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0236] These program instructions 41 may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing the program instructions 41 that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0237] It should be noted that the various methods, apparatuses, electronic devices, computer-readable storage media, computer program products, etc. described above may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0238] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0239] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acquire timbre feature data about the target user and initial audio data about the audio to be synthesized; the initial audio data is speech audio generated by the target user's associated user in response to the audio content of the audio to be synthesized, and the initial audio data is composed of multiple phoneme segments; The initial audio data of the audio to be synthesized is input into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and The initial audio data of the audio to be synthesized is input into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network. The timbre feature data of the target user is fused with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data, and the synthetic feature data is used to generate synthetic speech audio for the target user.

2. The method according to claim 1, characterized in that, The step of fusing the timbre feature data of the target user with the predicted fundamental frequency data and predicted duration data of each phoneme segment to obtain synthetic feature data includes: The initial audio data is segmented into phonemes to obtain multiple corresponding phoneme segments; Each of the phoneme segments is input into a pre-trained feature encoding network to obtain phoneme feature data for each of the phoneme segments output by the feature encoding network. According to the order of each phoneme segment, the phoneme feature data, predicted fundamental frequency data and predicted duration data corresponding to each phoneme segment are sequentially fused to obtain the fused feature data corresponding to each phoneme segment. The timbre feature data of the target user is fused with the fusion feature data of each phoneme segment to generate synthetic feature data for the audio to be synthesized.

3. The method according to claim 2, characterized in that, After obtaining the fusion feature data corresponding to each of the phoneme segments, the method further includes: The fusion feature data of each phoneme segment is segmented into single frames according to the prediction duration of the corresponding phoneme segment to obtain multiple single-frame fusion feature data corresponding to each phoneme segment. The step of fusing the timbre feature data of the target user with the fusion feature data of each phoneme segment to generate synthetic feature data for the audio to be synthesized includes: According to the order of the single-frame fusion feature data, the single-frame fusion feature data is sequentially fused with the timbre feature data of the target user to generate synthetic feature data for the audio to be synthesized.

4. The method according to claim 2, characterized in that, The synthesized feature data is used to characterize the Mel spectrum features of the audio to be synthesized, and the synthesized speech audio of the target user is a speech waveform file of the corresponding media format; The generation of synthesized speech audio for the target user includes: The Mel spectrum features are input into a pre-trained feature decoding network to obtain Mel spectrum data for the audio to be synthesized, which is output by the feature decoding network. The Mel spectrum data of the audio to be synthesized is converted into a waveform to obtain a speech waveform file corresponding to the media format of the synthesized speech audio.

5. The method according to claim 1, characterized in that, Obtain vocal characteristic data about the target user, including: Acquire initial audio data for the target user and a text sequence for the audio to be synthesized; the text sequence is used to express the audio content of the audio to be synthesized. The initial audio data of the target user is subjected to timbre extraction to obtain timbre feature data specific to the target user.

6. The method according to claim 1, characterized in that, Obtain initial audio data about the audio to be synthesized, including: From a preset audio database, an audio data set that matches the text sequence of the audio to be synthesized is determined; the audio data set includes multiple audio data, and each audio data is a voice audio of an associated user corresponding to the target user expressing the text sequence; In the audio data set, the target audio data with the highest similarity to the timbre feature data of the target user is identified, and the target audio data is used as the initial audio data for the audio to be synthesized.

7. The method according to claim 1, characterized in that, The steps include: inputting the initial audio data of the audio to be synthesized into a pre-trained fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and inputting the initial audio data of the audio to be synthesized into a pre-trained phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network, comprising: From the initial audio data of the audio to be synthesized, obtain the timbre feature data and phoneme feature data for each phoneme segment; The timbre feature data and phoneme feature data of each phoneme segment are input into the fundamental frequency prediction network to obtain the predicted fundamental frequency data for each phoneme segment output by the fundamental frequency prediction network; and The timbre feature data and phoneme feature data of each phoneme segment are input into the phoneme duration prediction network to obtain the predicted duration data for each phoneme segment output by the phoneme duration prediction network.

8. The method according to claim 1, characterized in that, Before inputting the initial audio data of the audio to be synthesized into the pre-trained fundamental frequency prediction network, the method further includes: Acquire multiple training audio samples for the training object; each training audio sample is based on multiple phoneme segments. Phoneme extraction and timbre extraction are performed on multiple training audio samples respectively to obtain feature data for each training audio sample; the feature data includes phoneme feature data and timbre feature data. The feature data of each training audio sample is input into the feature prediction network to be trained to obtain the predicted feature data for each training audio sample output by the feature prediction network to be trained; the predicted feature data includes the predicted fundamental frequency data or the predicted duration data for each phoneme segment in the training audio sample. Adjust the network parameters of the feature prediction network to be trained according to the predicted feature data to obtain the pre-trained feature prediction network. The pre-trained feature prediction network includes either the fundamental frequency prediction network or the phoneme duration prediction network. The fundamental frequency prediction network is used to predict the fundamental frequency of the initial audio data of the audio to be synthesized, and outputs the predicted fundamental frequency data for each phoneme segment in the initial audio data. The phoneme duration prediction network is used to predict the duration of the initial audio data of the audio to be synthesized, and outputs the predicted duration data for each phoneme segment in the initial audio data.

9. The method according to claim 8, characterized in that, After obtaining the pre-trained feature prediction network, the following is also included: Obtain audio data about the target object; Phoneme extraction or timbre extraction is performed on the audio data of the target object to obtain target phoneme feature data or target timbre feature data; The network parameters of the fundamental frequency prediction network are adjusted according to the target phoneme feature data to obtain the corresponding adjusted fundamental frequency prediction network; or, the network parameters of the phoneme duration prediction network are adjusted according to the target timbre feature data to obtain the corresponding adjusted phoneme duration prediction network.

10. A server, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the speech synthesis method as described in any one of claims 1 to 9.

11. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by the server's processor, it enables the server to perform the speech synthesis method as described in any one of claims 1 to 9.

Images