Speech synthesis method and device and electronic equipment

Abstract

The invention discloses a speech synthesis method and device and electronic equipment, and relates to the technical field of speech processing. The method comprises the steps of obtaining a singing audio and a to-be-synthesized text of a target object; determining a text vector of the text to be synthesized and an audio vector of the singing audio; based on the text vector and the audio vector, generating a target speech spectrum by using a target speech synthesis model; wherein when the target speech synthesis model generates the target speech spectrum, target speech features emphasizing speaking attribute features of the target object can be extracted through a feature extraction module, and singing attribute features of the target object are weakened.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, device and electronic device. Background Art

[0002] Text to speech (TTS) technology is a typical generative artificial intelligence technology. At present, the effect of speech synthesis depends largely on the reference speech data of the target object and the speech data used to train the speech synthesis model. For example, if the reference speech data or the speech data used to train the speech synthesis model is singing audio data, the synthesized speech will sound like singing and have unnatural intonation, which will bring a bad experience to users. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application provide a speech synthesis method, device and electronic device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a speech synthesis method, comprising:

[0006] Obtain the target object's singing audio and the text to be synthesized;

[0007] Determine a text vector of the text to be synthesized and an audio vector of the singing audio;

[0008] Based on the text vector and the audio vector, a target speech synthesis model is used to generate a target speech spectrum;

[0009] Among them, when generating the target speech spectrum, the target speech synthesis model can extract the target speech features focusing on the speaking attribute features of the target object through the feature extraction module, and weaken the singing attribute features of the target object.

[0010] In some embodiments, before generating a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector, the method further includes:

[0011] Obtain singing audio samples and text samples to be synthesized;

[0012] Determine a text vector sample of a text sample to be synthesized and an audio vector sample of a singing audio sample;

[0013] Based on the text vector samples and the audio vector samples, the initial speech synthesis model is used to generate speech spectrum samples;

[0014] Based on the speech spectrum samples and the specific speech classification model, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

[0015] In some embodiments, the specific voice classification model includes a first voice classification model and a second voice classification model; the first voice classification model is used to classify the voiceprint features of the speaker, and the second voice classification model is used to classify the singing attribute features and the speaking attribute features in the voiceprint features;

[0016] Based on the speech spectrum samples and the specific speech classification model, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model, including:

[0017] Inputting the speech spectrum sample into a first speech classification model to obtain a first loss value between a first prediction result and a first actual result of the first speech classification model;

[0018] Inputting the speech spectrum sample into a second speech classification model to obtain a second loss value between a second predicted result and a second actual result of the second speech classification model;

[0019] Based on the first loss value and the second loss value, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

[0020] In some embodiments, based on the first loss value and the second loss value, updating the parameters of the initial speech synthesis model to obtain the target speech synthesis model includes:

[0021] Negating the second loss value to obtain a third loss value having a gradient direction opposite to that of the second loss value;

[0022] Based on the first loss value and the third loss value, updating the parameters of the initial speech synthesis model to obtain a target speech synthesis model;

[0023] Among them, the third loss value is used to guide the parameters of the initial speech synthesis model to update in the direction of weakening the influence of the speaker's singing attribute characteristics.

[0024] In some embodiments, before inputting the speech spectrum sample into the second speech classification model and obtaining a second loss value between a second prediction result and a second actual result of the second speech classification model, the method further includes:

[0025] Obtain singing dry voice corpus and speaking voice corpus;

[0026] The singing dry voice corpus is used as the positive sample and the speaking voice corpus is used as the negative sample. Based on a specific singing judgment threshold, the parameters of the initial second speech classification model are adjusted to obtain the second speech classification model.

[0027] In some embodiments, determining a text vector of a text to be synthesized includes:

[0028] Converting the text to be synthesized into a first phoneme sequence;

[0029] Performing serialization processing on the first phoneme sequence to obtain a second phoneme sequence;

[0030] Based on the second phoneme sequence, a text vector of the text to be synthesized is determined.

[0031] In some embodiments, determining a text vector of the to-be-synthesized text based on the second phoneme sequence includes:

[0032] Allocating duration to each phoneme in the second phoneme sequence to obtain a third factor sequence with duration marks;

[0033] The third factor sequence is encoded to obtain the text vector of the text to be synthesized.

[0034] In some embodiments, the speech synthesis method further comprises:

[0035] The target speech spectrum is upsampled to obtain a speech time domain waveform corresponding to the target speech spectrum.

[0036] In a second aspect, an embodiment of the present application provides a speech synthesis device, including:

[0037] An acquisition module, used to acquire the singing audio of the target object and the text to be synthesized;

[0038] A determination module, used to determine a text vector of the text to be synthesized and an audio vector of the singing audio;

[0039] A generation module, used for generating a target speech spectrum using a target speech synthesis model based on a text vector and an audio vector;

[0040] Among them, when generating the target speech spectrum, the target speech synthesis model can extract the target speech features focusing on the speaking attribute features of the target object through the feature extraction module, and weaken the singing attribute features of the target object.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store executable data instructions; when the processor is used to execute the executable data instructions stored in the memory, the speech synthesis method provided in the first aspect is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method provided in the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of the composition of voiceprint vector information provided for related technologies;

[0045] Figure 2 One of the flowcharts of a speech synthesis method provided in an embodiment of the present application;

[0046] Figure 3 A second flowchart of a speech synthesis method provided in an embodiment of the present application;

[0047] Figure 4 A third flowchart of a speech synthesis method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a speech synthesis principle provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0052] It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0053] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant technical knowledge is first introduced as follows.

[0054] In recent years, speech synthesis or voice cloning technology for specific persons has been widely used in applications such as navigation and short video production. With the rapid development of generative artificial intelligence technology, speech synthesis technology for specific persons has made great progress and can generate speech with high naturalness and similarity based on shorter target object speech materials.

[0055] As technology advances, users' demands and expectations for speech synthesis are also increasing. In some scenarios, people want to hear a voice that "speaks as well as singing." For example, when a driver hears a favorite song while driving, he can set the voice of his in-car voice assistant to the timbre of the song, and the voice he hears when interacting with it is the type of voice he likes.

[0056] In the related technology, speech synthesis technology based on a small number of samples includes the following two:

[0057] (1) Few-shot solution based on speaker adaptation: A small amount of speech data and text data of the target object is used to fine-tune the pre-trained speech synthesis model to obtain a speech synthesis model tailored to the timbre of the target object.

[0058] (2) Zero-shot scheme based on speaker embedding prompt: When the number of reference speech of the target object is small, the target object’s voiceprint vector is extracted to guide the generation from text tokens to speech spectra.

[0059] However, when performing speech synthesis based on the above two speech synthesis technologies, it is necessary to extract the voiceprint vector of the target object to fine-tune the speech synthesis model or infer the speech spectrum (such as Mel Spectrum). Voiceprint vector is a set of vectors representing the speaker's voiceprint features extracted by a deep neural network, which contains as many distinguishing features of the speaker's voice attributes as possible and as little information related to the speech content as possible.

[0060] Human voiceprint is one of the important biological characteristics. For each person, their voiceprint is unique, including the structure of the vocal organs and the habits of each person using the vocal organs. Figure 1As shown in the figure, the pronunciation of people when speaking and singing is different. Therefore, the voiceprint vector extracted from the dry voice of the singing voice contains parameters of the pronunciation organs as well as the parameters of the pronunciation method when singing. When the extracted singing audio vector and the text vector are used for speech synthesis, the generated speech spectrum must contain the pronunciation method of singing. Even based on the duration and rhythm prediction results of the text, the generated speech spectrum will sound unnatural after being converted into a speech time domain waveform.

[0061] In order to improve the above-mentioned defects existing in the related technology, the embodiments of the present application provide a speech synthesis method, device and electronic device, which can generate a target speech spectrum based on a target speech synthesis model, extract target speech features that focus on the speaking attribute features of the target object through a feature extraction module, and weaken the singing attribute features of the target object, so that the generated target speech spectrum is converted into a speech time domain waveform without sounding unnatural, thereby improving the user experience.

[0062] The following is an illustrative introduction to the speech synthesis method, device and electronic device provided in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.

[0063] Figure 2 One of the flow charts of a speech synthesis method provided in an embodiment of the present application is as follows: Figure 2 As shown, the method includes:

[0064] S201, obtaining the singing audio of the target object and the text to be synthesized.

[0065] It should be noted that the target object may be an entity that has the ability to sing or has recorded singing audio. This entity may be a person, a virtual singer, or an existing audio file.

[0066] It should be noted that the singing audio can be an audio recording recorded or generated by the target object, containing the melody and / or lyrics. This audio recording is a resource that can be obtained or used, and can be used for music production, audio synthesis or other music-related activities.

[0067] It should be noted that the text to be synthesized is a text resource for synthesis with the singing audio of the target object, so as to convert the text to be synthesized into a target speech or audio book with the speaking attributes of the target object.

[0068] In some embodiments, a recording may be performed while the target object is singing to obtain the target object's singing audio.

[0069] In some embodiments, the target subject's singing audio may be obtained from a music platform, social media, or related websites.

[0070] In some embodiments, user-defined text to be synthesized may be obtained.

[0071] In some embodiments, based on user needs, texts that meet the user needs can be selected from various public resources as the text to be synthesized.

[0072] S202, determining the text vector of the text to be synthesized and the audio vector of the singing audio.

[0073] It should be noted that the text vector of the text to be synthesized may be a latent vector, which is a high-dimensional vector representation extracted from the text to be synthesized and capable of capturing the potential semantic features or patterns of the text to be synthesized.

[0074] In some embodiments, a text vector of the text to be synthesized can be generated based on a deep learning model.

[0075] In some embodiments, the text to be synthesized may be first converted into phonemes, and then the phonemes may be serialized and encoded to obtain a text vector of the text to be synthesized.

[0076] It should be noted that the audio vector of the singing audio can be a voiceprint representation vector, which can reflect the voiceprint characteristics of the target object individual, including characteristics such as sound quality, timbre, pitch, rhythm and melody.

[0077] In some embodiments, an audio vector for singing audio may be generated based on a deep learning model.

[0078] In some embodiments, audio vectors of singing audio may be generated based on conventional feature extraction algorithms.

[0079] S203. Generate a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector.

[0080] When generating the target speech spectrum, the target speech synthesis model can extract target speech features that focus on the speaking attribute features of the target object through a feature extraction module, and weaken the singing attribute features of the target object.

[0081] It should be noted that the target speech synthesis model can be a model for speech synthesis that is pre-built and trained. Moreover, the model has a feature extraction module. When the target speech synthesis model generates a target speech spectrum based on the text vector of the text to be synthesized and the audio vector of the singing audio of the target object, the feature extraction module can extract the target speech features that focus on the speaking attribute features of the target object and weaken the singing attribute features of the target object, so that the generated target speech spectrum will not sound unnatural after being converted into a speech time domain waveform, thereby improving the user experience.

[0082] It should be noted that the target speech synthesis model may be a model based on deep learning. The structure of the target speech synthesis model is not specifically limited in the embodiments of the present application.

[0083] In some embodiments, the text vector of the text to be synthesized and the audio vector of the singing audio of the target object can be input into the target speech synthesis model, so that the target speech synthesis model can extract the target speech features that focus on the speaking attribute features of the target object based on the input audio vector through the feature extraction module, and weaken the singing attribute features of the target object, and then generate a target speech spectrum with significant speaking attribute features of the target object based on the extracted target speech features and the input text vector.

[0084] In some embodiments, the target speech spectrum generated by the target speech synthesis model may be a Mel Spectrum.

[0085] It can be understood that the speech synthesis method provided in the embodiment of the present application generates a target speech spectrum by using a target speech synthesis model based on the text vector of the text to be synthesized and the audio vector of the singing audio of the target object. Moreover, when the target speech synthesis model generates the target speech spectrum, the feature extraction module extracts the target speech features that focus on the speaking attribute features of the target object and weakens the singing attribute features of the target object, so that the finally generated target speech spectrum can better reflect the acoustic characteristics of the target object when speaking, such as intonation, speaking speed and timbre; and, compared with the singing audio of the target object, the generated target speech spectrum will significantly reduce or eliminate the features related to singing, such as melody, rhythm and pitch, so that the finally generated target speech spectrum, after being converted into a speech time domain waveform, sounds more like the target object is speaking rather than singing, without the unnatural sense of intonation, thereby improving the user experience.

[0086] In some embodiments, before generating a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector, the method further includes:

[0087] Obtain singing audio samples and text samples to be synthesized;

[0088] Determine a text vector sample of the text sample to be synthesized and an audio vector sample of the singing audio sample;

[0089] Based on the text vector sample and the audio vector sample, generating a speech spectrum sample using an initial speech synthesis model;

[0090] Based on the speech spectrum samples and the specific speech classification model, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

[0091] In an embodiment of the present application, multiple singing audio samples of multiple different objects and multiple text samples to be synthesized can be obtained, and each singing audio sample can be converted into a corresponding audio vector sample, and each text sample to be synthesized can be converted into a corresponding text vector sample, and then any audio vector sample and any text vector sample can be input as a group into an initial speech synthesis model to obtain a speech spectrum sample generated by the initial speech synthesis model. Since multiple audio vector samples and multiple text vector samples can constitute multiple combinations, the initial speech synthesis model can be used to generate multiple speech spectrum samples, and then based on the multiple speech spectrum samples and the specific speech classification model, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

[0092] It should be noted that the specific speech classification model may be a model pre-built based on a deep learning model for classifying the speech features of a speaker, wherein the speech features may include voiceprint features, and the voiceprint features may include singing attribute features and speaking attribute features.

[0093] It should be noted that the voiceprint feature is a unique acoustic attribute of each person when pronouncing a word, which can be used to identify or classify the speaker.

[0094] It should be noted that the initial speech synthesis model may be a model that is initially constructed based on a deep learning model and has not yet been trained or adjusted.

[0095] In some embodiments, speech spectrum samples can be input into a specific speech classification model to obtain a loss value between the predicted result and the actual result of the specific speech classification model, and then based on the loss value, the parameters of the initial speech synthesis model are guided to be updated in the direction of weakening the influence of the speaker's singing attribute characteristics, so that the target speech synthesis model finally obtained can extract the speaking attribute features that focus on the speaker through the feature extraction module and weaken the singing attribute characteristics of the speaker during speech synthesis.

[0096] It can be understood that the embodiment of the present application generates speech spectrum samples by using an initial speech synthesis model based on text vector samples and audio vector samples, and then updates the parameters of the initial speech synthesis model based on the speech spectrum samples and a specific speech classification model, thereby effectively obtaining a target speech synthesis model that can weaken the singing attribute characteristics of the speaker.

[0097] In some embodiments, the specific voice classification model includes a first voice classification model and a second voice classification model; the first voice classification model is used to classify the voiceprint features of the speaker, and the second voice classification model is used to classify the singing attribute features and the speaking attribute features in the voiceprint features;

[0098] The updating of the parameters of the initial speech synthesis model based on the speech spectrum sample and the specific speech classification model to obtain the target speech synthesis model includes:

[0099] Inputting the speech spectrum sample into the first speech classification model to obtain a first loss value between a first prediction result and a first actual result of the first speech classification model;

[0100] Inputting the speech spectrum sample into the second speech classification model to obtain a second loss value between a second prediction result and a second actual result of the second speech classification model;

[0101] Based on the first loss value and the second loss value, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

[0102] In the embodiment of the present application, the parameters of the initial speech synthesis model can be updated based on the speech spectrum sample and the first speech classification model and the second speech classification model to obtain the target speech synthesis model. The first speech classification model can classify the voiceprint features of the speaker, and the second speech classification model can classify the singing attribute features and the speaking attribute features in the voiceprint features.

[0103] In some embodiments, a speech spectrum sample can be input into a first speech classification model to obtain a first loss value between a first prediction result of the first speech classification model and a first actual result. The first prediction result is a voiceprint feature classification judgment made by the first speech classification model based on the input speech spectrum sample, the first actual result is a known true classification label corresponding to the input speech spectrum sample, and the first loss value is calculated based on the difference between the first prediction result and the first actual result. The first loss value can be used to measure the prediction performance of the first speech classification model under a given input.

[0104] In some embodiments, the speech spectrum sample can be input into the second speech classification model to obtain a second loss value between the second prediction result of the second speech classification model and the second actual result. The second prediction result is the classification judgment of the singing attribute features and the speaking attribute features made by the second speech classification model based on the input speech spectrum sample, the second actual result is a known true classification label corresponding to the input speech spectrum sample, and the second loss value is calculated based on the difference between the second prediction result and the second actual result. The second loss value can be used to measure the prediction performance of the second speech classification model under a given input.

[0105] In some embodiments, the gradients of the first loss value and the second loss value relative to the parameters of the initial speech synthesis model can be determined respectively, and then the parameters of the initial speech synthesis model can be updated based on the two gradients to obtain the target speech synthesis model.

[0106] It can be understood that since the purpose of the embodiment of the present application is to enable the target speech synthesis model to weaken the singing attribute characteristics of the speaker and focus on extracting the speaking attribute characteristics of the speaker, it can be determined that the gradient of the first loss value relative to the initial speech synthesis model parameters is a positive gradient, and the gradient of the second loss value relative to the initial speech synthesis model parameters is a negative gradient, so that the parameters of the initial speech synthesis model are updated in the direction of weakening the influence of the speaker's singing attribute characteristics, and are updated in the direction of focusing on extracting the speaker's speaking attribute characteristics.

[0107] It can be understood that the embodiment of the present application updates the parameters of the initial speech synthesis model by combining the first loss value of the first speech classification model and the second loss value of the second speech classification model, which can effectively guide the parameter update direction of the initial speech synthesis model, so that the target speech synthesis model obtained after the parameter update can weaken the singing attribute characteristics of the speaker and focus on extracting the speaking attribute characteristics of the speaker.

[0108] In some embodiments, updating the parameters of the initial speech synthesis model based on the first loss value and the second loss value to obtain the target speech synthesis model includes:

[0109] Negating the second loss value to obtain a third loss value having a gradient direction opposite to that of the second loss value;

[0110] Based on the first loss value and the third loss value, updating the parameters of the initial speech synthesis model to obtain the target speech synthesis model;

[0111] Among them, the third loss value is used to guide the parameters of the initial speech synthesis model to be updated in the direction of weakening the influence of the speaker's singing attribute characteristics.

[0112] It should be noted that since the purpose of the embodiment of the present application is to enable the target speech synthesis model to weaken the singing attribute characteristics of the speaker, the gradient of the second loss value relative to the parameters of the initial speech synthesis model is a negative gradient. Therefore, the second loss value can be negatively processed (that is, the opposite of the second loss value is taken or the second loss value is multiplied by -1) to obtain a third loss value with the opposite gradient direction to the second loss value. The third loss value can then guide the parameters of the initial speech synthesis model to be updated in the direction of weakening the influence of the speaker's singing attribute characteristics.

[0113] In some embodiments, based on the first loss value and the third loss value, an optimization algorithm (such as a stochastic gradient descent algorithm) can be used to update the parameters of the initial speech synthesis model to obtain a target speech synthesis model that can weaken the singing attribute characteristics of the speaker.

[0114] It can be understood that the embodiment of the present application obtains a third loss value with a gradient direction opposite to that of the second loss value by negating the second loss value of the second speech classification model, and then combines the third loss value and the first loss value of the first speech classification model to update the parameters of the initial speech synthesis model. This can more effectively guide the parameter update direction of the initial speech synthesis model, so that the target speech synthesis model obtained after the parameter update can weaken the singing attribute characteristics of the speaker and focus on extracting the speaking attribute characteristics of the speaker.

[0115] In some embodiments, before inputting the speech spectrum sample into the second speech classification model to obtain a second loss value between a second prediction result and a second actual result of the second speech classification model, the method further includes:

[0116] Obtain singing dry voice corpus and speaking voice corpus;

[0117] The singing dry voice corpus is used as a positive sample, the speaking voice corpus is used as a negative sample, and based on a specific singing judgment threshold, the parameters of the initial second voice classification model are adjusted to obtain the second voice classification model.

[0118] In an embodiment of the present application, multiple singing dry voice corpora and speaking voice corpora of multiple different objects can be obtained, and then the singing dry voice corpora are used as positive samples, and the speaking voice corpora are used as negative samples to train the initial second voice classification model. During the training process, the parameters of the initial second voice classification model are continuously adjusted to minimize the value of the loss function and improve the accuracy of the classification of the initial second voice classification model. Moreover, after the training is completed, the parameters of the second voice classification model are further adjusted based on a specific singing judgment threshold to further improve the accuracy of the second voice classification model in classifying singing attribute features and speaking attribute features.

[0119] It should be noted that the singing dry voice corpus can be pure human voice singing audio without any spatial property post-processing (such as reverberation, delay, etc.). These audios usually only contain the original singing sound, without background music, harmony or other sound effects. When training the initial second voice classification model, the singing dry voice corpus is used as a positive sample, which can enable the initial second voice classification model to learn and recognize the sound of the singing category.

[0120] It should be noted that the spoken voice corpus can be pure human voice audio in daily communication. These audios can contain various intonations, rhythms and emotions, but do not contain singing or music elements. When training the initial second voice classification model, the spoken voice corpus is used as a negative sample, which can enable the initial second voice classification model to learn and recognize non-singing sounds.

[0121] It should be noted that the initial second speech classification model can learn the difference between singing and speaking by comparing positive samples and negative samples, and classify them accordingly.

[0122] It should be noted that after the initial second voice classification training is completed, the model will output a probability value or score, indicating the possibility that the input audio belongs to the singing category. In order to convert the model output into a specific classification result, the probability value or score output by the model can be compared with a specific singing determination threshold. When it is higher than the specific singing determination threshold, the input audio is classified as singing audio; otherwise, the input audio is classified as speaking audio.

[0123] It should be noted that the specific song determination threshold can be determined based on factors such as the performance of the second speech classification model and actual needs, and the present embodiment does not specifically limit this. For example, the specific song determination threshold is 0.70, 0.80 or 0.85.

[0124] It can be understood that the embodiment of the present application uses singing dry voice corpus as positive samples and speaking voice corpus as negative samples, and based on a specific singing judgment threshold, adjusts the parameters of the initial second voice classification model to obtain the second voice classification model, which can improve the accuracy of the second voice classification model in classifying the singing attribute features and speaking attribute features in the voiceprint features.

[0125] In some embodiments, determining the text vector of the text to be synthesized includes:

[0126] Converting the text to be synthesized into a first phoneme sequence;

[0127] Performing serialization processing on the first phoneme sequence to obtain a second phoneme sequence;

[0128] Based on the second phoneme sequence, a text vector of the text to be synthesized is determined.

[0129] In an embodiment of the present application, a phoneme annotation tool can be used to convert the text to be synthesized into a first phoneme sequence, and then the first phoneme sequence is serialized, including but not limited to phoneme encoding, phoneme merging or splitting, and phoneme order adjustment, etc., to obtain a second phoneme sequence, and then a fixed-size vector (i.e., an embedding vector) is assigned to each phoneme in the second phoneme sequence to convert discrete phoneme symbols into points in a continuous vector space, thereby capturing the semantic and phonetic similarities between phonemes, thereby obtaining a text vector that can reflect the potential semantic features or patterns of the text to be synthesized.

[0130] For example, Figure 3 A second flow chart of a speech synthesis method provided in an embodiment of the present application is as follows: Figure 3 As shown, the method includes:

[0131] S301, obtaining the singing audio of the target object and the text to be synthesized.

[0132] S302: Convert the text to be synthesized into a first phoneme sequence.

[0133] S303: Perform serialization processing on the first phoneme sequence to obtain a second phoneme sequence.

[0134] S304. Determine a text vector of the text to be synthesized based on the second phoneme sequence.

[0135] S305: Determine the audio vector of the singing audio.

[0136] S306. Generate a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector.

[0137] It should be noted that there is no sequential order relationship between S302~S304 and S305, and they can be executed simultaneously, or in sequence according to the above steps, or S305 can be executed first, and then S302~S304 can be executed in sequence. The embodiment of the present application does not specifically limit this execution order.

[0138] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0139] It can be understood that the embodiment of the present application converts the text to be synthesized into a first phoneme sequence, and then serializes the first phoneme sequence to obtain a second phoneme sequence. Based on the second phoneme sequence, a text vector that can reflect the potential semantic features or patterns of the text to be synthesized can be effectively obtained, thereby providing strong technical support for subsequent speech synthesis.

[0140] In some embodiments, determining the text vector of the text to be synthesized based on the second phoneme sequence includes:

[0141] Allocating duration to each phoneme in the second phoneme sequence to obtain a third factor sequence with duration marks;

[0142] The third factor sequence is encoded to obtain a text vector of the text to be synthesized.

[0143] In an embodiment of the present application, the second phoneme sequence can be parsed first to determine the identity and order of each factor in the second phoneme sequence, and then a pre-trained duration prediction model or a rule-based duration allocation method can be used to assign a specific pronunciation duration (usually in milliseconds or frames) to each phoneme to obtain a third factor sequence with a duration mark, in which each phoneme is attached with a corresponding pronunciation duration, and then the third phoneme sequence with a duration mark is encoded to convert the third phoneme sequence with a duration mark into a text vector suitable for speech synthesis.

[0144] It should be noted that the duration prediction model can predict the pronunciation duration of each phoneme based on text features (such as phonemes, syllables and word lengths, etc.), prosodic features (such as pauses, stress and intonation, etc.) and possible linguistic knowledge.

[0145] For example, Figure 4 A third flow chart of a speech synthesis method provided in an embodiment of the present application is as follows: Figure 4 As shown, the method includes:

[0146] S401, obtaining the singing audio of the target object and the text to be synthesized.

[0147] S402: Convert the text to be synthesized into a first phoneme sequence.

[0148] S403: Perform serialization processing on the first phoneme sequence to obtain a second phoneme sequence.

[0149] S404: Allocate duration to each phoneme in the second phoneme sequence to obtain a third factor sequence with duration marks.

[0150] S405: Encode the third factor sequence to obtain a text vector of the text to be synthesized.

[0151] S406: Determine the audio vector of the singing audio.

[0152] S407: Generate a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector.

[0153] It should be noted that there is no sequential order relationship between S402~S405 and S406, and they can be executed simultaneously, or executed sequentially according to the above steps, or S406 can be executed first, and then S402~S405 can be executed sequentially. The embodiment of the present application does not specifically limit this execution order.

[0154] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0155] It can be understood that the embodiment of the present application helps to avoid pauses and uneven speaking speed in speech synthesis by allocating the duration of each phoneme in the second phoneme sequence, making the synthesized speech more fluent and coherent in terms of intonation, rhythm, etc., and more in line with the natural characteristics of human language.

[0156] In some embodiments, the speech synthesis method further comprises:

[0157] The target speech spectrum is upsampled to obtain a speech time domain waveform corresponding to the target speech spectrum.

[0158] In the embodiment of the present application, after the target speech spectrum is generated by using the target speech synthesis model, the target speech spectrum can be upsampled to obtain a speech time domain waveform corresponding to the target speech spectrum.

[0159] It should be noted that upsampling can be to increase the sampling rate of the signal, that is, to increase the number of samples per unit time. In the spectrum domain, upsampling means inserting new spectrum components between the zero points of the spectrum, or increasing the resolution of the spectrum by some interpolation method.

[0160] In some embodiments, after upsampling the target speech spectrum, an inverse Fourier transform may be performed to convert the target speech spectrum into the time domain to obtain a speech time domain waveform corresponding to the target speech spectrum.

[0161] It should be noted that the speech time domain waveform can show the changes of the speech signal over time and is the expression form of the speech signal in the time domain.

[0162] For example, Figure 5 A schematic diagram of a speech synthesis principle provided in an embodiment of the present application is shown in FIG. Figure 5As shown, the phonemes corresponding to the text to be synthesized can be input into the text segmentation module, so that the text segmentation module can serialize the phonemes corresponding to the text to be synthesized and obtain a serialized phoneme sequence; at the same time, the phonemes corresponding to the text to be synthesized can also be input into the duration prediction module, so that the duration prediction module can allocate duration to each phoneme and obtain a phoneme sequence with a duration mark; then the serialized phoneme sequence and the phoneme sequence with the duration mark are combined and input into the encoder for encoding to obtain a text vector of the text to be synthesized; then the text vector of the text to be synthesized and the audio vector of the singing audio (the audio vector is obtained by extracting features of the singing audio based on the voiceprint extraction model) are input into the speech synthesis model to obtain a speech spectrum generated by the speech synthesis model; finally, the speech spectrum generated by the speech synthesis model is input into the vocoder module, so that the vocoder module converts the speech spectrum into a speech time domain waveform. Among them, the speech synthesis model is obtained by adjusting parameters based on the loss value of the speaker classifier (corresponding to the first speech classification model described above) and the negative loss value of the singing-speaking classifier (corresponding to the second speech classification model described above). It has the ability to decouple the singing pronunciation method in the voiceprint without increasing additional computing resources and delays.

[0163] It can be understood that the embodiment of the present application uses a speaker classifier and a singing-speaking classifier to guide the speech synthesis model to update parameters during the enhanced training of the speech synthesis model, so that the speech spectrum inferred by the speech synthesis model can retain the speaker's speaking attribute characteristics but weaken the singing attribute characteristics. Among them, the loss value of the speaker classifier propagates the positive gradient to ensure that the speech spectrum has voiceprint distinguishability and retains the speaking attribute characteristics of the target object; the loss value of the singing-speaking classifier propagates the negative gradient to make the speech spectrum contain as few singing attribute characteristics as possible.

[0164] In some embodiments, the text to be synthesized can be converted into phonemes, which are then serialized by the text segmentation module, and then the duration distribution of the phoneme sequence is processed under the guidance of the duration prediction module to make it consistent with the natural way of human speech. For different languages, the steps involved in converting the text to be synthesized into phonemes are different. The conversion of Chinese text can include the regularization of symbols and numbers and the conversion of polyphones.

[0165] In some embodiments, the encoder module may encode a phoneme sequence into a latent vector, which may represent a priori or a posteriori probability distribution of the speech spectrum.

[0166] In some embodiments, the speech synthesis model can be a generative model based on a normalizing flow model or a stable diffusion model (a text-to-image generative model based on a diffusion model). For the voice cloning task, the audio embedding of the singing audio (as a reference audio) can be used as prompt information to guide the speech synthesis model to generate a speech spectrum (such as a Mel spectrum) including the timbre of the target object singing audio.

[0167] In some embodiments, the voiceprint extraction model may be a speaker recognition model that has been pre-trained using a large amount of sample data and has the ability to extract high-quality voiceprint representation vectors.

[0168] In some embodiments, the vocoder module may upsample the speech spectrum to generate a speech time domain waveform. The vocoder module may upsample the speech spectrum based on any one of MelGAN, HifiGAN, WaveNet, Griff-Lim and iFFT to generate a speech time domain waveform.

[0169] It should be noted that the MelGAN method takes the Mel spectrum map as input, upsamples it through transposed convolution (deconvolution), and gradually increases the length of the output sequence until it matches the time resolution of the original speech waveform; the HifiGAN method takes the Mel spectrum map as input, upsamples and expands it through transposed convolution and multi-receptive field fusion modules until it outputs a sequence that matches the original speech waveform; the WaveNet method uses a convolutional neural network to model audio signals. It regards the audio signal as a series of discrete sample points and predicts the value of the next sample point through a convolutional neural network. In this way, the entire audio signal can be gradually generated; the Griff-Lim method is a speech waveform reconstruction method with a known amplitude spectrum and an unknown phase spectrum. It generates a phase spectrum through iteration and reconstructs the speech waveform with the known amplitude spectrum and the calculated phase spectrum; the iFFT method is a mathematical method for converting frequency domain signals into time domain signals. In speech synthesis, iFFT can be used to convert frequency domain features such as Mel spectrum into time domain waveforms.

[0170] It should be noted that the speaker classifier and the singing-speaking classifier are two voiceprint attribute feature decoupling auxiliary modules, which are used to enhance the training of the speech synthesis model.

[0171] In some embodiments, the model structures of the speaker classifier, the singing-speaking classifier, and the voiceprint extraction model may be the same or different, and this embodiment of the present application does not specifically limit this.

[0172] In some embodiments, the speaker classifier may be trained based on a cosine similarity scoring method.

[0173] In some embodiments, various singing dry voice corpora can be used as positive samples and normal speaking voice corpora can be used as negative samples, and parameters of the singing-speaking classifier can be adjusted based on a specific singing judgment threshold.

[0174] It should be noted that the speech synthesis method provided in the embodiment of the present application can be applied to scenarios including but not limited to vehicle-mounted and Internet of Things (IOT). Normal speech synthesis can be achieved based on a small amount of singing audio samples, providing users with a more demanding voice interaction experience. The embodiment of the present application can at least bring the following beneficial effects: (1) There is no need to obtain the target object's voice data in both singing and speaking modes. Since it is difficult to find the original speaking voice for most online singers in practice, it is even more difficult to achieve a certain amount of high-quality speech; (2) The speaker classifier and the singing-speaking classifier can use pre-trained models, and do not need to participate in the speech synthesis model training to increase the difficulty of convergence; (3) The speech synthesis model does not require a special decoupling module in the inference stage, does not increase the consumption of computing resources, and does not cause additional delays.

[0175] The speech synthesis device provided in the embodiment of the present application is described below. The speech synthesis device described below and the speech synthesis method described above can be referenced to each other.

[0176] Figure 6 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 610, a determination module 620 and a generation module 630; wherein:

[0177] An acquisition module 610 is used to acquire the singing audio of the target object and the text to be synthesized;

[0178] A determination module 620, used to determine the text vector of the text to be synthesized and the audio vector of the singing audio;

[0179] A generating module 630, configured to generate a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector;

[0180] When generating the target speech spectrum, the target speech synthesis model can extract target speech features that focus on the speaking attribute features of the target object through a feature extraction module, and weaken the singing attribute features of the target object.

[0181] The speech synthesis device provided in the embodiment of the present application generates a target speech spectrum by using a target speech synthesis model based on a text vector of the text to be synthesized and an audio vector of the singing audio of the target object. Moreover, when the target speech synthesis model generates the target speech spectrum, the feature extraction module extracts the target speech features that focus on the speaking attribute features of the target object and weakens the singing attribute features of the target object, so that the finally generated target speech spectrum can better reflect the acoustic characteristics of the target object when speaking, such as intonation, speaking speed and timbre; and, compared with the singing audio of the target object, the generated target speech spectrum will significantly reduce or eliminate the features related to singing, such as melody, rhythm and pitch, so that the finally generated target speech spectrum, after being converted into a speech time domain waveform, sounds more like the target object is speaking rather than singing, without the unnatural sense of intonation, thereby improving the user experience.

[0182] In some embodiments, the acquisition module 610 is further used to: acquire a singing audio sample and a text sample to be synthesized;

[0183] The determination module 620 is further used to: determine the text vector sample of the text sample to be synthesized, and the audio vector sample of the singing audio sample;

[0184] The generating module 630 is further used to: generate speech spectrum samples using an initial speech synthesis model based on the text vector samples and the audio vector samples;

[0185] The device also includes: an updating module, which is used to update the parameters of the initial speech synthesis model based on the speech spectrum samples and the specific speech classification model to obtain the target speech synthesis model.

[0186] In some embodiments, the specific voice classification model includes a first voice classification model and a second voice classification model; the first voice classification model is used to classify the voiceprint features of the speaker, and the second voice classification model is used to classify the singing attribute features and the speaking attribute features in the voiceprint features;

[0187] The update module includes:

[0188] A first obtaining unit, configured to input the speech spectrum sample into the first speech classification model to obtain a first loss value between a first prediction result and a first actual result of the first speech classification model;

[0189] A second obtaining unit, configured to input the speech spectrum sample into the second speech classification model to obtain a second loss value between a second prediction result and a second actual result of the second speech classification model;

[0190] The third obtaining unit is used to update the parameters of the initial speech synthesis model based on the first loss value and the second loss value to obtain the target speech synthesis model.

[0191] In some embodiments, the third acquisition unit is also used to: negate the second loss value to obtain a third loss value with a gradient direction opposite to that of the second loss value; update the parameters of the initial speech synthesis model based on the first loss value and the third loss value to obtain the target speech synthesis model; wherein the third loss value is used to guide the update of the parameters of the initial speech synthesis model in the direction of weakening the influence of the speaker's singing attribute characteristics.

[0192] In some embodiments, the acquisition module 610 is further used to: acquire singing dry voice corpus and speaking voice corpus;

[0193] The device also includes: a parameter adjustment module, which is used to use the singing dry voice corpus as a positive sample and the speaking voice corpus as a negative sample, and based on a specific singing judgment threshold, adjust the parameters of the initial second voice classification model to obtain the second voice classification model.

[0194] In some embodiments, the determination module 620 includes:

[0195] A conversion unit, used for converting the text to be synthesized into a first phoneme sequence;

[0196] a fourth obtaining unit, configured to perform serialization processing on the first phoneme sequence to obtain a second phoneme sequence;

[0197] A determination unit is used to determine a text vector of the text to be synthesized based on the second phoneme sequence.

[0198] In some embodiments, the determination unit is further used to: allocate duration to each phoneme in the second phoneme sequence to obtain a third factor sequence carrying a duration mark; encode the third factor sequence to obtain a text vector of the text to be synthesized.

[0199] In some embodiments, the device further includes a sampling module for performing upsampling processing on the target speech spectrum to obtain a speech time domain waveform corresponding to the target speech spectrum.

[0200] It should be noted here that the above-mentioned speech synthesis device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned speech synthesis method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0201] Figure 7A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the executable data instructions stored in the memory 730 to execute part or all of the steps in the speech synthesis method provided in the above embodiments.

[0202] In addition, the executable data instructions stored in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0203] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, some or all of the steps in the speech synthesis method provided in the above embodiments are implemented.

[0204] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute some or all of the steps in the speech synthesis method provided in the above embodiments.

[0205] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0206] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0207] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0210] The above description is merely an optional embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A speech synthesis method, comprising: Obtain the target object's singing audio and the text to be synthesized; Determine a text vector of the text to be synthesized and an audio vector of the singing audio; Based on the text vector and the audio vector, generating a target speech spectrum using a target speech synthesis model; When generating the target speech spectrum, the target speech synthesis model can extract target speech features that focus on the speaking attribute features of the target object through a feature extraction module, and weaken the singing attribute features of the target object.

2. The speech synthesis method according to claim 1, before generating a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector, the method further comprises: Obtain singing audio samples and text samples to be synthesized; Determine a text vector sample of the text sample to be synthesized and an audio vector sample of the singing audio sample; Based on the text vector sample and the audio vector sample, generating a speech spectrum sample using an initial speech synthesis model; Based on the speech spectrum samples and the specific speech classification model, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

3. The speech synthesis method according to claim 2, wherein the specific speech classification model comprises a first speech classification model and a second speech classification model; the first speech classification model is used to classify the voiceprint features of the speaker, and the second speech classification model is used to classify the singing attribute features and the speaking attribute features in the voiceprint features; The updating of the parameters of the initial speech synthesis model based on the speech spectrum sample and the specific speech classification model to obtain the target speech synthesis model includes: Inputting the speech spectrum sample into the first speech classification model to obtain a first loss value between a first prediction result and a first actual result of the first speech classification model; Inputting the speech spectrum sample into the second speech classification model to obtain a second loss value between a second prediction result and a second actual result of the second speech classification model; Based on the first loss value and the second loss value, the parameters of the initial speech synthesis model are updated to obtain the target speech synthesis model.

4. The speech synthesis method according to claim 3, wherein the updating of the parameters of the initial speech synthesis model based on the first loss value and the second loss value to obtain the target speech synthesis model comprises: Negating the second loss value to obtain a third loss value having a gradient direction opposite to that of the second loss value; Based on the first loss value and the third loss value, updating the parameters of the initial speech synthesis model to obtain the target speech synthesis model; Among them, the third loss value is used to guide the parameters of the initial speech synthesis model to be updated in the direction of weakening the influence of the speaker's singing attribute characteristics.

5. The speech synthesis method according to claim 3 or 4, before inputting the speech spectrum sample into the second speech classification model to obtain a second loss value between a second prediction result and a second actual result of the second speech classification model, the method further comprises: Obtain singing dry voice corpus and speaking voice corpus; The singing dry voice corpus is used as a positive sample, the speaking voice corpus is used as a negative sample, and based on a specific singing judgment threshold, the parameters of the initial second voice classification model are adjusted to obtain the second voice classification model.

6. The speech synthesis method according to any one of claims 1 to 4, wherein determining the text vector of the text to be synthesized comprises: Converting the text to be synthesized into a first phoneme sequence; Performing serialization processing on the first phoneme sequence to obtain a second phoneme sequence; Based on the second phoneme sequence, a text vector of the text to be synthesized is determined.

7. The speech synthesis method according to claim 6, wherein determining the text vector of the text to be synthesized based on the second phoneme sequence comprises: Allocating duration to each phoneme in the second phoneme sequence to obtain a third factor sequence with duration marks; The third factor sequence is encoded to obtain a text vector of the text to be synthesized.

8. The speech synthesis method according to any one of claims 1 to 4, further comprising: The target speech spectrum is upsampled to obtain a speech time domain waveform corresponding to the target speech spectrum.

9. A speech synthesis device, comprising: An acquisition module, used to acquire the singing audio of the target object and the text to be synthesized; A determination module, used to determine the text vector of the text to be synthesized and the audio vector of the singing audio; A generating module, configured to generate a target speech spectrum using a target speech synthesis model based on the text vector and the audio vector; When generating the target speech spectrum, the target speech synthesis model can extract target speech features that focus on the speaking attribute features of the target object through a feature extraction module, and weaken the singing attribute features of the target object.

10. An electronic device comprising: A memory for storing executable data instructions; A processor, configured to implement the speech synthesis method according to any one of claims 1 to 8 when executing the executable data instructions stored in the memory.

Images