A speech synthesis method, an electronic device, and a computer-readable storage medium

Abstract

The application discloses a speech synthesis method, an electronic device and a medium. A plurality of phoneme sequences corresponding to a to-be-processed text are obtained. Phoneme features corresponding to each phoneme in the phoneme sequences are extracted. The phoneme features corresponding to each phoneme are decoupled in tone to obtain decoupled phoneme features. Audio conforming to the tone of a target person is obtained according to the decoupled phoneme features and tone features of the target person. Decoupling the phoneme features reduces the interference caused by the original personal tone features. When audio conforming to the tone of the target person needs to be generated, there is no need to obtain a large amount of audio data with the tone features of the target person for training. Only the tone features of the target person need to be extracted from a small amount of audio data with the tone features of the target person. According to the decoupled phoneme features and the tone features of the target person, audio conforming to the tone of the target person can be obtained, and the cost of tone transfer is reduced.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speech synthesis method, electronic device, and computer-readable storage medium. Background Technology

[0002] Speech synthesis is the process of generating corresponding audio from input text. Speech synthesis technology has wide applications in voice assistants, chatbots, audiobooks, and virtual humans. With the surge in demand for personalized voices, voice transfer technology is becoming increasingly important. Voice transfer can generate audio with the voice of a target person, such as a specific broadcaster or celebrity.

[0003] Traditional methods require recording a large amount of speech from the target person and training the model on a pre-trained base model to learn the target person's tone, rhythm, accent, and other timbre characteristics. Since everyone has a unique timbre, generating audio with a specific person's timbre typically requires recording several hours of their speech to train the model. The training process requires a large number of training samples, and the more samples there are, the higher the computational cost of training the model, resulting in high costs for timbre transfer.

[0004] In summary, during the process of realizing this application, the inventors discovered that the prior art has at least the problem of high cost of timbre transfer. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a speech synthesis method, electronic device, and computer-readable storage medium that can obtain audio with the timbre of a target person based on a small amount of training data, thereby reducing the cost of timbre transfer. The specific solution is as follows:

[0006] In a first aspect, this application discloses a speech synthesis method, including:

[0007] Obtain multiple phoneme sequences corresponding to the text to be processed;

[0008] The phoneme features corresponding to each phoneme are decoupled by timbre to obtain the decoupled phoneme features;

[0009] Based on the decoupled phoneme features and the timbre features of the target person, an audio recording that matches the timbre of the target person is obtained.

[0010] Optionally, the step of decoupling the timbre of the phoneme features corresponding to each phoneme to obtain the decoupled phoneme features includes:

[0011] Statistical analysis of the average phoneme features corresponding to the same phoneme in different phoneme sequences;

[0012] Based on the average phoneme features corresponding to each phoneme, the phoneme features of each phoneme sequence are adjusted to obtain the decoupled phoneme features.

[0013] Optionally, adjusting the phoneme features of each phoneme sequence based on the average phoneme features corresponding to each phoneme to obtain the decoupled phoneme features includes:

[0014] The mean squared error is calculated by performing a mean squared error operation on the target average phoneme feature corresponding to the target phoneme and the target phoneme feature corresponding to the target phoneme in the target phoneme sequence to obtain the decoupled target phoneme feature; wherein, the target phoneme is any one of all phonemes; and the target phoneme sequence is any one of all phoneme sequences.

[0015] Optionally, the step of decoupling the timbre of the phoneme features corresponding to each phoneme to obtain the decoupled phoneme features includes:

[0016] A timbre discriminator is used to identify the individual timbre features contained in the phoneme features;

[0017] Based on the actual timbre features corresponding to the text to be processed and the personal timbre features identified by the timbre discriminator, the timbre recognition loss is determined.

[0018] The timbre recognition loss is backpropagated to the synthesizer used for gradient inversion to obtain an updated synthesizer;

[0019] The updated synthesizer is used to extract the target phoneme features corresponding to each phoneme in the phoneme sequence, and the target phoneme features are used as the decoupled phoneme features.

[0020] Optionally, after performing timbre decoupling on the phoneme features corresponding to each phoneme to obtain the decoupled phoneme features, the method further includes:

[0021] The decoupled phoneme features are stored in a database;

[0022] Upon receiving a timbre adjustment instruction, the decoupled phoneme features are retrieved from the database; wherein the timbre adjustment instruction carries the target timbre features.

[0023] Based on the decoupled phoneme features and the target timbre features, an audio signal that matches the target timbre is obtained.

[0024] Optionally, after performing timbre decoupling on the phoneme features corresponding to each phoneme to obtain the decoupled phoneme features, the method further includes:

[0025] The decoupled phoneme features are stored in the cloud.

[0026] If the decoupled phoneme features do not exist in the local database, the decoupled phoneme features are obtained from the cloud.

[0027] Optionally, obtaining the audio that matches the timbre of the target person based on the decoupled phoneme features and the timbre features of the target person includes:

[0028] The decoupled phoneme features and the timbre features of the target person are spliced ​​together to obtain phoneme features carrying the timbre of the target person;

[0029] The decoder is used to decode the phoneme features carrying the target person's timbre into a frame sequence;

[0030] The frame sequence is processed using a vocoder to obtain audio that matches the voice of the target person.

[0031] Optionally, obtaining the audio that matches the timbre of the target person based on the decoupled phoneme features and the timbre features of the target person includes:

[0032] The decoupled phoneme features and the timbre features of the target person are fused using a deep learning neural network model to obtain an audio that matches the timbre of the target person.

[0033] Secondly, this application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the aforementioned disclosed speech synthesis method.

[0036] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned disclosed speech synthesis method.

[0037] In this application, multiple phoneme sequences corresponding to the text to be processed are obtained; phoneme features corresponding to each phoneme in the phoneme sequence are extracted; for a text to be processed, the personal timbre features contained in the phoneme features can be fully extracted through the multiple phoneme sequences corresponding to the text to be processed. In order to reduce the interference caused by personal timbre features to the phoneme features, the phoneme features corresponding to each phoneme can be decoupled to obtain decoupled phoneme features. The decoupled phoneme features effectively filter out personal timbre features, and based on the decoupled phoneme features and the timbre features of the target person, audio that is more consistent with the timbre of the target person can be obtained. In this technical solution, the phoneme features are decoupled, reducing the interference caused by the original personal timbre features. When it is necessary to generate audio that matches the timbre of the target person, it is not necessary to obtain a large amount of audio data with the timbre features of the target person for training. It is only necessary to extract the timbre features of the target person from a small amount of audio data with the timbre features of the target person. Based on the decoupled phoneme features and the timbre features of the target person, audio that matches the timbre of the target person can be quickly obtained. Audio that matches the target person's voice can be obtained with a small amount of audio data, effectively reducing the cost of voice transfer. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A system framework diagram applicable to a speech synthesis method provided in this application;

[0040] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of this application;

[0041] Figure 3 A schematic diagram of a neural network model provided in this application;

[0042] Figure 4 This application provides a structural diagram of an electronic device. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Traditionally, generating audio that matches a person's voice requires acquiring a large amount of audio data containing that voice. This data is then used to train a model, allowing the model to learn the voice's characteristics. While this ensures accuracy, it also results in high computational costs for training the model and thus high costs associated with voice transfer.

[0045] Therefore, embodiments of this application provide a speech synthesis method, an electronic device, and a computer-readable storage medium. After acquiring multiple phoneme sequences corresponding to the text to be processed and extracting the phoneme features corresponding to each phoneme in the phoneme sequences, the phoneme features corresponding to each phoneme can be decoupled by timbre to obtain decoupled phoneme features. The decoupled phoneme features effectively filter out personal timbre features. Based on the decoupled phoneme features and the timbre features of the target person, audio that is more consistent with the timbre of the target person can be obtained. Through this processing method, text can be converted into audio that matches the timbre of the target person using only a small amount of audio from the target person.

[0046] Figure 1 This is a system framework diagram applicable to a speech synthesis method provided in this application. Figure 1 This example uses the combination of the ECAPA-TDNN (Emphasized Channel Attention Propagation and Aggregation in TDNN) and FastSpeech network models. The ECAPA-TDNN model extracts timbre features from audio segments, while the FastSpeech model generates a phoneme sequence of the text to be processed. The phoneme embedding layer and FFT block layer of the FastSpeech model process the phoneme sequence to obtain phoneme features. A loss function, Loss, can be added to the FastSpeech network model. mse The phoneme features are decoupled based on the loss function, thereby effectively filtering out the personal timbre features contained in the phoneme features. Figure 1 The symbol H is used in the middle. pho This represents the decoupled phoneme features. The Length Regulator layer of the FastSpeech network model can concatenate the decoupled phoneme features with the timbre features of the target person. After processing by other layers in the FastSpeech network model, the final output is an audio that matches the timbre of the target person.

[0047] For ease of description, the ECAPA-TDNN network model and the FastSpeech network model will be used as examples in the following content. However, it should be noted that... Figure 1The ECAPA-TDNN and FastSpeech network models are merely illustrative examples. In practical applications, other types of network models can also be used to perform the above operations. For example, a Long Short-Term Memory (LSTM) network model can be used to extract timbre features, and an attention model can be used to perform the acquisition of phoneme features, decoupling of phoneme features, and synthesis of the decoupled phoneme features and the timbre features of the target person.

[0048] The speech synthesis method provided in this application can be applied to a variety of scenarios, such as audiobooks, celebrity voice assistants, and simulating the speech of relatives.

[0049] Figure 2 A flowchart illustrating a speech synthesis method provided in an embodiment of this application. See also... Figure 2 As shown, the speech synthesis method may include the following steps:

[0050] S201: Obtain multiple phoneme sequences corresponding to the text to be processed.

[0051] The text to be processed can be plain text information that needs to be converted into audio. The text to be processed can be an article, a news report, or a traffic report, etc. In this embodiment, the specific form of the text to be processed is not limited.

[0052] In practical applications, users often want the text to be processed to be read aloud using the voice of a person they like. Since there are many different people, for easy differentiation, the person the user likes can be called the target person. Based on this need, the text to be processed needs to be converted into audio that matches the voice of the target person.

[0053] S202: Extract the phoneme features corresponding to each phoneme in the phoneme sequence.

[0054] A phoneme sequence often contains phoneme features corresponding to multiple phonemes. Different phoneme sequences may contain the same or different phonemes. Taking a single phoneme as an example, the phoneme features corresponding to that phoneme can be extracted from all phoneme sequences.

[0055] In this embodiment of the application, an audio signal matching the timbre of the target person is obtained by synthesizing phoneme features and the timbre features of the target person.

[0056] Taking the FastSpeech network model as an example, when converting text information into audio, the FastSpeech network model first converts the text information into a phoneme sequence, then converts the phoneme sequence into phoneme features, and finally generates audio based on the phoneme features. To better match the user's listening habits, the phoneme features automatically include timbre features that conform to the normal speaking habits of people. However, when converting text information into audio that matches the timbre of a target person, the timbre features automatically included in the phoneme features can become interfering factors. Therefore, in this embodiment, to fully mine the personal timbre features contained in the phoneme features, multiple phoneme sequences corresponding to the text to be processed can be obtained simultaneously, and the phoneme features corresponding to the multiple phoneme sequences can be compared to mine the personal timbre features contained in the phoneme features.

[0057] S203: Decouple the timbre of the phoneme features corresponding to each phoneme to obtain the decoupled phoneme features.

[0058] In the embodiments of this application, there are multiple ways to achieve timbre decoupling. One feasible approach is to mine the individual timbre features contained in each phoneme and statistically analyze the average phoneme features corresponding to the same phoneme in different phoneme sequences; the average phoneme features can reflect the common features of phonemes. Based on the average phoneme features corresponding to each phoneme, the phoneme features of each phoneme sequence are adjusted to obtain the decoupled phoneme features.

[0059] Decoupling phoneme features mainly involves retaining the common features among phoneme features and eliminating individual timbre features.

[0060] The network model's processing of phoneme sequences involves multiple iterations, each of which calculates the average phoneme feature corresponding to the phoneme. In this embodiment, during each iteration, the phoneme features of the current iteration can be adjusted based on the average phoneme features obtained from the previous iteration, thereby filtering out individual timbre features contained in the phoneme features of the current iteration.

[0061] To reduce the interference of individual timbre characteristics on phoneme characteristics, timbre decoupling can be performed on the phoneme characteristics corresponding to each phoneme to obtain decoupled phoneme characteristics. In the embodiments of this application, the phoneme characteristics can be adjusted by calculating the mean square error, thereby achieving the purpose of timbre decoupling.

[0062] The processing of phoneme features for each phoneme sequence is similar. Taking any one of the phoneme sequences, i.e. the target phoneme sequence, as an example, the phoneme features of the target phoneme sequence contain features corresponding to multiple phonemes. The following is a detailed introduction using any one of the phonemes, i.e. the target phoneme, as an example.

[0063] In a practical implementation, the mean squared error can be calculated between the target average phoneme feature corresponding to the target phoneme and the target phoneme feature corresponding to the target phoneme in the target phoneme sequence to obtain the decoupled target phoneme feature.

[0064] For example, let the phoneme set P = [p1, p2, ..., p...] N It contains N phonemes, where p i Let X represent the i-th phoneme, where 1 ≤ i ≤ N. The text to be processed contains multiple phoneme sequences that can be used for training. i Let X represent the i-th phoneme sequence. i =[x i1 ,x i2 ,···], where x ij This represents the j-th phoneme in the i-th phoneme sequence, i.e., each x ij As a phoneme, x ij ∈P.

[0065] In the k-th epoch of model training, each phoneme sequence, after passing through the phoneme embedding layer and encoder layer of the FastSpeech network model, yields the phoneme feature H. pho H k,i This represents the phonetic feature corresponding to the i-th phoneme sequence. The phoneme x represents the phoneme extracted by the model at the k-th epoch. ij The phoneme features. The sequence length of each phoneme sequence is consistent with the sequence length of its corresponding phoneme feature.

[0066] Because x ij Since each phoneme is a single phoneme, by looking up correspondences, each phoneme can find its corresponding phoneme features in different phoneme sequences. Averaging the phoneme features corresponding to each phoneme across different phoneme sequences yields the result. That is, the phoneme p i The average phoneme features of the FastSpeech network model at the k-th epoch.

[0067] In the (k+1)th epoch of model training, the phoneme sequence X i After passing through the phoneme embedding layer and encoder layer of the FastSpeech network model, the phoneme features can be obtained.

[0068] In addition to the original loss function of the FastSpeech network model, another loss function is added to decouple the phoneme features from their timbre. The added loss function can be set according to the Mean Square Error (MSE) calculation method.

[0069] The purpose of adding a loss function is to retain only the common features of the same phoneme from different speakers, while filtering out the individual timbre features of different speakers. The expression for the loss function is as follows:

[0070]

[0071] The meaning of this loss function is to let each phoneme x ij All can approximate its overall expression

[0072] Decoupled phoneme features H pho It retains only the collective characteristics of phonemes, removing the speaker's individual timbre characteristics.

[0073] Besides the MSE method described above, the idea of ​​Generative Adversarial Network (GAN) can also be used to ensure that the obtained phoneme features do not contain personal timbre features. In the embodiments of this application, a timbre discriminator can be used to identify the personal timbre features contained in the phoneme features; based on the actual timbre features corresponding to the text to be processed and the personal timbre features identified by the timbre discriminator, a timbre recognition loss is determined; the timbre recognition loss is backpropagated to a synthesizer used for gradient inversion to obtain an updated synthesizer; the updated synthesizer is used to extract the target phoneme features corresponding to each phoneme in the phoneme sequence, and the target phoneme features are used as the decoupled phoneme features.

[0074] In practical implementations, timbre recognition loss is often presented as a loss function. After determining the timbre recognition loss function, the gradient of each network layer in the neural network model formed by the timbre discriminator and synthesizer can be calculated using the backpropagation algorithm. Based on this gradient, the neural network parameters can be updated, thus obtaining a trained neural network model. To ensure that the phoneme features output by the synthesizer can remove the speaker's personal timbre, the gradient of the synthesizer can be inverted. Based on the inverted gradient, the network parameters of the synthesizer can be updated, so that the phoneme features extracted by the updated synthesizer do not contain the speaker's personal timbre.

[0075] Figure 3 This is a schematic diagram of a neural network model provided in an embodiment of this application. Specifically, let the phoneme feature H pho The output is fed to the discriminator neural network, in H pho The discriminator is trained using speaker timbre labels, therefore the discriminator has the ability to distinguish H... pho The speaker's vocal timbre ability; phoneme sequence to H pho The network layer acts as a synthesizer, determining how phonemes are generated (H). pho .like Figure 3Figure (a) shows the training process of a typical classification neural network, which calculates the loss function using speaker timbre labels and the output of the discriminator. disc By using the backpropagation algorithm to calculate the gradient of each network layer, the parameters of the neural network can be updated to train the model. Figure 3 Figure (b) is a schematic diagram of a neural network model for filtering out personal vocal characteristics provided in an embodiment of this application. Utilizing the idea of ​​Generative Adversarial Networks (GANs), the loss function can be optimized. disc The gradient is inverted during backpropagation to the synthesizer. Gradient inversion can be achieved by multiplying the original gradient by -1, which makes the discriminator and synthesizer work against each other, allowing H to... pho Without the speaker's timbre, the updated synthesizer is used to extract the target phoneme features corresponding to each phoneme in the phoneme sequence. Since the target phoneme features no longer carry the individual's timbre, they can be used as the decoupled phoneme features.

[0076] S204: Based on the decoupled phoneme features and the timbre features of the target person, obtain an audio that matches the timbre of the target person.

[0077] The decoupled phoneme features effectively filter out individual timbre characteristics. Based on the decoupled phoneme features and the timbre characteristics of the target person, an audio that is more in line with the timbre of the target person can be obtained.

[0078] To obtain the vocal timbre characteristics of a target person, it is only necessary to extract them from an audio frequency band containing the target person's vocal timbre within a few minutes. In the embodiments of this application, an audio segment of the target person can be obtained; the audio segment can be analyzed using a vocal timbre extraction network model to extract the vocal timbre characteristics of the target person.

[0079] There are various types of timbre extraction network models, such as the ECAPA-TDNN network model and the Long Short-Term Memory network model.

[0080] In practical applications, any one of the network models can be chosen to extract the timbre features of an audio segment. For example, the ECAPA-TDNN network model can be used to extract the timbre features of an audio segment.

[0081] In this application embodiment, the method of obtaining audio that matches the timbre of the target person based on the decoupled phoneme features and the timbre features of the target person is not limited. One feasible implementation is to concatenate the decoupled phoneme features and the timbre features of the target person to obtain phoneme features carrying the timbre of the target person; use a decoder to decode the phoneme features carrying the timbre of the target person into a frame sequence; and use a vocoder to process the frame sequence to obtain audio that matches the timbre of the target person.

[0082] Both the decoupled phoneme features and the target person's timbre features are presented in vector form. Concatenating the decoupled phoneme features and the target person's timbre features can be done by directly adding or multiplying them.

[0083] In addition to the methods described above, deep learning neural network models can also be used to fuse the decoupled phoneme features and the timbre features of the target person to obtain audio that matches the timbre of the target person.

[0084] In this application, multiple phoneme sequences corresponding to the text to be processed are obtained, and phoneme features corresponding to each phoneme in the phoneme sequences are extracted. For a text to be processed, the personal timbre features contained in the phoneme features can be fully extracted through the multiple phoneme sequences corresponding to the text. In order to reduce the interference of personal timbre features on phoneme features, timbre decoupling can be performed on the phoneme features corresponding to each phoneme to obtain decoupled phoneme features. The decoupled phoneme features effectively filter out personal timbre features. Based on the decoupled phoneme features and the timbre features of the target person, audio that is more consistent with the timbre of the target person can be obtained. In this technical solution, the phoneme features are decoupled, reducing the interference caused by the original personal timbre features. When it is necessary to generate audio that matches the timbre of the target person, it is not necessary to obtain a large amount of audio data with the timbre features of the target person for training. It is only necessary to extract the timbre features of the target person from a small amount of audio data with the timbre features of the target person, and synthesize them based on the decoupled phoneme features and the timbre features of the target person to quickly obtain audio that matches the timbre of the target person. Audio that matches the target person's voice can be obtained with a small amount of audio data, effectively reducing the cost of voice transfer.

[0085] The above description uses obtaining audio that matches the target person's voice as an example. In practical applications, to meet different voice switching needs, a voice adjustment mechanism can be set up. Furthermore, to improve the efficiency of voice adjustment, after obtaining the decoupled phoneme features, these features can be stored in a local database. This allows for direct retrieval of the decoupled phoneme features from the local database when subsequent voice adjustment is needed.

[0086] In practical implementation, when a user needs to switch the voice playback timbre, they can select the desired target timbre on the human-computer interaction interface of the audio playback device. For the audio playback device, clicking the confirmation option corresponding to the target timbre on the human-computer interaction interface is equivalent to inputting a timbre adjustment command. The audio playback device can pre-store the timbre characteristics corresponding to different timbres. After the user selects a target timbre, the target timbre characteristics can be obtained by querying the stored timbre characteristics.

[0087] When an audio playback device receives a timbre adjustment command, it can retrieve the decoupled phoneme features from the database; based on the decoupled phoneme features and the target timbre features, it can obtain audio that matches the target timbre.

[0088] Considering the limited storage space of local databases in practical applications and the susceptibility to accidental deletion of information, this embodiment of the application allows the decoupled phoneme features to be stored in the cloud. If the decoupled phoneme features are not present in the local database, they can be retrieved from the cloud.

[0089] In this embodiment, a timbre adjustment mechanism can be set to meet different timbre switching needs. By storing the decoupled phoneme features, the decoupled phoneme features can be directly called when performing timbre switching, eliminating the need to perform timbre decoupling operations again and improving the efficiency of timbre switching.

[0090] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be considered as any limitation on the scope of this application.

[0091] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech synthesis method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a server.

[0092] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0093] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and video data 223, etc., and the storage method can be temporary storage or permanent storage.

[0094] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive video data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the speech synthesis method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include various video data collected by the electronic device 20.

[0095] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the speech synthesis method steps disclosed in any of the foregoing embodiments.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0097] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] The above provides a detailed description of the speech synthesis method, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech synthesis method characterized by, The method comprises the following steps: obtaining a plurality of phoneme sequences corresponding to a to-be-processed text; the to-be-processed text comprises phoneme sequences corresponding to a plurality of different speakers; using each phoneme sequence of the to-be-processed text as training data, determining a decoupling loss function according to the fact that, in each iteration, each phoneme feature is close to the average feature of the corresponding phoneme in all phoneme sequences in the last iteration, and the phoneme sequences and the phoneme positions are superimposed, and using the decoupling loss function and the original loss function of the network model as the loss function used in the model training process; the network model is trained on each phoneme sequence for multiple iterations based on the loss function, and the phoneme features in the current iteration are adjusted according to the average phoneme features in the last iteration: in the kth iteration of model training, the phoneme features of each phoneme sequence in the kth iteration are determined by the network model, the corresponding phoneme features of each phoneme in different phoneme sequences are determined according to the consistent relationship between the length of the phoneme sequence and the length of the corresponding phoneme feature sequence, and the average value of the phoneme features of each phoneme in different phoneme sequences is taken as the average phoneme feature of the phoneme in the kth iteration; extracting the phoneme features corresponding to each phoneme in the phoneme sequence: each phoneme sequence comprises a plurality of phonemes, the length of each phoneme sequence is consistent with the length of the corresponding phoneme feature sequence, and the corresponding phoneme features of each phoneme in different phoneme sequences are determined by searching for the corresponding relationship for each phoneme; performing voice decoupling on the phoneme features corresponding to each phoneme to obtain decoupled phoneme features; obtaining an audio in the voice of a target person according to the decoupled phoneme features and the voice features of the target person; wherein the voice decoupling on the phoneme features corresponding to each phoneme comprises: statistically obtaining the average phoneme features corresponding to the same phoneme in different phoneme sequences; performing mean square difference operation on the target average phoneme features corresponding to the target phoneme and the target phoneme features corresponding to the target phoneme in the target phoneme sequence to obtain the decoupled target phoneme features.

2. The speech synthesis method of claim 1, wherein, The voice decoupling on the phoneme features corresponding to each phoneme to obtain decoupled phoneme features comprises: adjusting the phoneme features of each phoneme sequence based on the average phoneme features corresponding to each phoneme to obtain decoupled phoneme features.

3. The speech synthesis method of claim 2, wherein, The target phoneme is any one of all phonemes; the target phoneme sequence is any one of all phoneme sequences.

4. The speech synthesis method of claim 1, wherein, The voice decoupling on the phoneme features corresponding to each phoneme to obtain decoupled phoneme features comprises: using a voice discriminator to identify the personal voice features contained in the phoneme features; determining a voice recognition loss based on the actual voice features corresponding to the to-be-processed text and the personal voice features identified by the voice discriminator; propagating the voice recognition loss back to a synthesizer for gradient negation to obtain an updated synthesizer; using the updated synthesizer to extract target phoneme features corresponding to each phoneme in the phoneme sequence, and taking the target phoneme features as decoupled phoneme features.

5. The speech synthesis method of claim 1, wherein, After the voice decoupling on the phoneme features corresponding to each phoneme to obtain decoupled phoneme features, the method further comprises the following steps: store the decoupled phoneme features to a local database; in the case of obtaining a timbre adjustment instruction, call the decoupled phoneme features from the local database; obtain audio in accordance with the decoupled phoneme features and target timbre features corresponding to the timbre adjustment instruction.

6. The speech synthesis method of claim 5, wherein, after the phoneme features corresponding to each phoneme are decoupled to obtain decoupled phoneme features, further comprising: store the decoupled phoneme features to the cloud; in the case that the decoupled phoneme features do not exist in the local database, obtain the decoupled phoneme features from the cloud.

7. The speech synthesis method of claim 1, wherein, the decoupled phoneme features and the timbre features of the target person are spliced to obtain phoneme features carrying the timbre of the target person; the phoneme features carrying the timbre of the target person are decoded into a frame sequence using a decoder; the frame sequence is processed using a vocoder to obtain audio in accordance with the timbre of the target person. the decoupled phoneme features and the timbre features of the target person are spliced to obtain phoneme features carrying the timbre of the target person; 8. The speech synthesis method of claim 1, wherein, a deep learning neural network model is used to fuse the decoupled phoneme features and the timbre features of the target person to obtain audio in accordance with the timbre of the target person. comprising:

9. An electronic device, comprising: a memory for saving a computer program; a processor for executing the computer program to implement the speech synthesis method of any one of claims 1 to 8. a memory for saving a computer program, which, when executed by a processor, implements the speech synthesis method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, ​

Images