Speech synthesis method and device, equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the steps that a prediction style sequence is acquired based on a voice text and a first audio clip, and the first audio clip is used for acquiring the speaking style of a first object; determining a target Mel spectrogram based on the second audio clip and the prediction style sequence; and determining predicted speech based on the target Mel spectrogram. According to the method, by obtaining the prediction style sequence, style migration can be more natural and richer based on the relationship between the speaking style of the first object and the phonemes of the voice text. According to the method, the predicted style sequence is firstly obtained, then the tone is obtained, and then the Mel spectrum is generated, so that the style and the tone can be effectively decoupled, and the problem of tone leakage is reduced. Therefore, the predicted voice synthesized by the voice synthesis method is accurate in tone quality and has good style expressive force.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] Cross-speaker style transfer speech synthesis refers to transferring the style of the first speaker to the timbre of the second speaker, so that the synthesized speech has the style of the first speaker and the timbre of the second speaker. However, current speech synthesis methods have certain limitations in dealing with the decoupling of timbre and style, resulting in timbre leakage and poor style expression, resulting in unsatisfactory timbre and style performance of the synthesized speech. Therefore, a new speech synthesis method is urgently needed. Summary of the invention

[0003] The present disclosure provides a speech synthesis method, apparatus, device and storage medium. The method can effectively decouple style and timbre, reducing the problem of timbre leakage. Therefore, the predicted speech synthesized by the speech synthesis method has accurate timbre and good style expression.

[0004] According to one aspect of an embodiment of the present disclosure, a speech synthesis method is provided, wherein the method comprises:

[0005] Based on the speech text and the first audio segment, obtaining a predicted style sequence, wherein the first audio segment is used to obtain a speaking style of a first subject, and the predicted style sequence is used to represent a relationship between the speaking style of the first subject and the phonemes of the speech text;

[0006] Determining a target mel-spectrogram based on a second audio segment and the predicted style sequence, wherein the second audio segment is used to obtain the timbre of a second object;

[0007] Based on the target mel-spectrogram, a predicted speech is determined, where the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

[0008] According to another aspect of an embodiment of the present disclosure, a speech synthesis device is provided, the device comprising:

[0009] an acquisition unit configured to acquire a predicted style sequence based on a speech text and a first audio segment, wherein the first audio segment is used to acquire a speaking style of a first subject, and the predicted style sequence is used to represent a relationship between the speaking style of the first subject and phonemes of the speech text;

[0010] A first determining unit is configured to determine a target mel-spectrogram based on a second audio segment and the predicted style sequence, wherein the second audio segment is used to obtain a timbre of a second object;

[0011] The second determination unit is configured to determine a predicted speech based on the target mel-spectrogram, where the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

[0012] In some embodiments, the acquisition unit is configured to convert the speech text into a phoneme sequence; perform style encoding on the first audio clip to obtain a prompt style sequence; input the phoneme sequence and the prompt style sequence into a style language model to obtain the predicted style sequence, and the style language model is used to capture the relationship between phonemes and styles through an attention mechanism.

[0013] In some embodiments, the acquisition unit is configured to normalize the first audio segment; acquire the normalized mel-spectrogram of the first audio segment; and input the mel-spectrogram of the first audio segment into a style encoder to obtain the prompt style sequence.

[0014] In some embodiments, the acquisition unit is further configured to align the first audio segment and the speech text in a time dimension so that the length of the morphemes in the speech text is consistent with the length of the corresponding speech segment in the first audio segment.

[0015] In some embodiments, the first determination unit is configured to encode the second audio segment to obtain a speech embedding representation of the second object; input the speech embedding representation and the predicted style sequence into an acoustic diffusion model to obtain the target Mel-spectrogram, and the acoustic diffusion model is used to gradually generate a Mel-spectrogram from noise using a back diffusion process.

[0016] In some embodiments, the second determination unit is configured to input the target mel-spectrogram into a waveform diffusion model to generate the predicted speech, and the waveform diffusion model is used to generate a high-fidelity audio signal by stepwise denoising using a reverse diffusion process.

[0017] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] one or more processors;

[0019] A memory for storing program codes executable by the processor;

[0020] The processor is configured to execute the program code to implement the above-mentioned speech synthesis method.

[0021] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned speech synthesis method.

[0022] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned speech synthesis method when executed by a processor.

[0023] The disclosed embodiment provides a speech synthesis solution, which can be more natural and rich in style transfer based on the relationship between the speaking style of the first subject and the phonemes of the speech text by obtaining a predicted style sequence. By first obtaining the predicted style sequence, then obtaining the timbre, and then generating a mel-spectrogram, the style and timbre can be effectively decoupled, reducing the timbre leakage problem. Therefore, the predicted speech synthesized by the above-mentioned speech synthesis method has accurate timbre and good style expression.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 It is a schematic diagram of an implementation environment of a speech synthesis method according to an exemplary embodiment.

[0027] Figure 2 The figure is a flow chart of a speech synthesis method according to an exemplary embodiment.

[0028] Figure 3 The figure is a flow chart of another speech synthesis method according to an exemplary embodiment.

[0029] Figure 4 It is a flowchart of a speech synthesis method provided according to an exemplary embodiment.

[0030] Figure 5 The present invention is a block diagram of a speech synthesis device according to an exemplary embodiment.

[0031] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0032] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the audio clips and voice texts involved in this disclosure are all obtained with full authorization.

[0035] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a speech synthesis method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0036] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application is used to play information stream data, such as multimedia data or live broadcast data. The application is associated with the server 102, and the server 102 provides background services to the terminal 101.

[0037] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0038] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0039] Figure 2 is a flow chart of a speech synthesis method according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0040] In step S201, a predicted style sequence is obtained based on the speech text and the first audio segment.

[0041] In the disclosed embodiment, the text content of the speech text is the same as the speech content of the synthesized speech to be generated. The first audio clip is used to obtain the speaking style of the first subject. In other words, the first audio clip is a basis for obtaining the speaking style of the first subject. Optionally, the speaking style includes various characteristics such as speaking speed, intonation, stress, pauses, etc. The role of the predicted style sequence is to represent the relationship between the speaking style of the first subject and the phonemes of the speech text.

[0042] Phonemes are the smallest units of speech, and different speaking styles will affect the performance of phonemes. For example, a certain style may make the pronunciation of a specific phoneme longer or shorter, or have special intonation changes at a specific phoneme.

[0043] In step S202, a target mel-spectrogram is determined based on the second audio segment and the predicted style sequence.

[0044] In the disclosed embodiment, the second audio clip is used to obtain the timbre of the second object. The timbre is the characteristic of the sound. Different people have different timbre characteristics. The timbre is determined by factors such as the frequency and harmonics of the sound. The second object can be the same as the first object or different from the first object.

[0045] Since the predicted style sequence represents the relationship between the speaking style of the first subject and the phonemes of the speech text, the target Mel-spectrogram is determined based on the timbre of the second subject represented by the second audio clip and the predicted style sequence. Among them, the Mel-spectrogram is a representation method that can reflect the energy distribution of sound at different frequencies. By combining the timbre of the second subject and the predicted style sequence, the determined target Mel-spectrogram can have both the unique timbre of the second subject and the speaking style of the first subject, thereby providing a basis for subsequent operations such as speech synthesis.

[0046] In step S203, the predicted speech is determined based on the target mel-spectrogram.

[0047] In the disclosed embodiment, the predicted speech can be obtained by decoding the target mel-spectrogram, wherein the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

[0048] The disclosed embodiment provides a speech synthesis solution, which can be more natural and rich in style transfer based on the relationship between the speaking style of the first subject and the phonemes of the speech text by obtaining a predicted style sequence. By first obtaining the predicted style sequence, then obtaining the timbre, and then generating a mel-spectrogram, the style and timbre can be effectively decoupled, reducing the timbre leakage problem. Therefore, the predicted speech synthesized by the above-mentioned speech synthesis method has accurate timbre and good style expression.

[0049] In some embodiments, obtaining a predicted style sequence based on the speech text and the first audio segment includes:

[0050] Converting spoken text into phoneme sequences;

[0051] Performing style encoding on the first audio clip to obtain a prompt style sequence;

[0052] The phoneme sequence and the prompt style sequence are input into the style language model to obtain the predicted style sequence. The style language model is used to capture the relationship between phonemes and styles through the attention mechanism.

[0053] By converting speech text into a phoneme sequence, the first audio clip is style-encoded to obtain a prompt style sequence, and then both are input into the style language model to obtain a predicted style sequence. The attention mechanism of the style language model can accurately capture the relationship between phonemes and styles, providing a key style basis for synthesizing speech with a specific style, which helps to achieve high-quality speech style conversion and synthesis.

[0054] In some embodiments, performing style encoding on the first audio clip to obtain a prompt style sequence includes:

[0055] Normalizing the first audio segment;

[0056] Obtain a normalized mel-spectrogram of the first audio segment;

[0057] The mel-spectrogram of the first audio segment is input into the style encoder to obtain a prompt style sequence.

[0058] The first audio clip is normalized to obtain its mel-spectrogram, and then input into the style encoder to obtain the prompt style sequence. This process helps to eliminate audio differences, accurately extract the style of the first audio clip, and provide reliable style guidance for subsequent speech processing, thereby achieving more accurate speech style conversion and synthesis.

[0059] In some embodiments, the method further comprises:

[0060] The first audio segment and the speech text are aligned in a time dimension so that the length of the morphemes in the speech text is consistent with the length of the corresponding speech segment in the first audio segment.

[0061] By aligning the first audio segment and the speech text in the time dimension, it can be ensured that the morphemes in the speech text are consistent with the length of the corresponding speech segment in the first audio segment, which is conducive to more accurately applying the speaking style of the first audio segment to the speech text, and providing a more accurate basis for the subsequent synthesis of speech with a specific style.

[0062] In some embodiments, determining a target mel-spectrogram based on the second audio segment and the predicted style sequence includes:

[0063] Encoding the second audio segment to obtain a speech embedding representation of the second object;

[0064] The speech embedding representation and the predicted style sequence are input into the acoustic diffusion model to obtain the target Mel-spectrogram. The acoustic diffusion model is used to gradually generate the Mel-spectrogram from the noise using the back-diffusion process.

[0065] A voice embedding representation is obtained by encoding a second audio segment, and then combined with a predicted style sequence and input into an acoustic diffusion model to determine a target mel spectrogram. By using the reverse diffusion process of the acoustic diffusion model, a mel spectrogram that contains both the timbre of the second object and reflects the style represented by the predicted style sequence can be generated, providing an accurate basis for synthesizing speech with a specific timbre and style, and improving the quality and expressiveness of speech synthesis.

[0066] In some embodiments, based on the target mel spectrogram, determining predicted speech includes:

[0067] Inputting the target mel spectrogram into a waveform diffusion model to generate predicted speech, where the waveform diffusion model is used to gradually denoise and generate a high-fidelity audio signal by using the reverse diffusion process.

[0068] By inputting the target mel spectrogram into the waveform diffusion model to generate predicted speech, and the waveform diffusion model gradually denoises and generates a high-fidelity audio signal by using the reverse diffusion process, which makes the synthesized predicted speech have higher sound quality and realism, providing a better speech synthesis experience for users.

[0069] The above Figure 2 shows a flowchart of a speech synthesis method of the present disclosure. The speech synthesis solution provided by the present disclosure will be further described below. Figure 3 is a flowchart of another speech synthesis method shown according to an exemplary embodiment. Refer to Figure 3 , which is executed by an electronic device and includes the following steps:

[0070] In step S301, convert the speech text into a phoneme sequence.

[0071] In the embodiments of the present disclosure, the speech text is a sentence or paragraph composed of a series of words. The text content of the speech text is the speech content of the synthesized speech to be generated. A phoneme is the smallest unit in speech and is the smallest speech unit divided according to the natural attributes of speech. The process of converting the speech text into a phoneme sequence refers to analyzing each word or syllable in the text, determining its corresponding phoneme composition, and arranging these phonemes in sequence according to the order of the text to form a sequence of phonemes. The phoneme sequence can provide a basic speech feature representation for subsequent speech processing tasks, such as speech synthesis and speech recognition.

[0072] For example, in English, phonemes can include vowels (such as "a", "e", "i", "o", "u") and consonants (such as "b", "c", "d", etc.). In Chinese, for the speech text "你好" (Hello), it may be converted into a phoneme sequence "n-i, h-ao".

[0073] In some embodiments, the electronic device converts the speech text into a phoneme sequence through a text-to-phoneme module. Optionally, the text-to-phoneme module is a G2P (Grapheme-to-Phoneme) module. The module is built based on the Transformer architecture. The module uses a large amount of text-phoneme pair data during training to ensure efficient phoneme conversion.

[0074] In step S302, style encoding is performed on the first audio segment to obtain a prompt style sequence.

[0075] In the disclosed embodiment, the first audio clip contains the sound information of the first subject speaking, which can reflect the speaking style of the first subject. The style coding extracts and analyzes various features in the audio through a specific algorithm and model to determine the style characteristics of the speaking. Optionally, the style characteristics include multiple aspects such as speaking speed, intonation, stress distribution, pause pattern, etc.

[0076] The prompt style sequence is a digital representation of the speaking style of the first subject. The prompt style sequence can reflect the speaking style of the first subject. The prompt style sequence can be used to guide other speech processing tasks in subsequent processing, such as synthesizing speech with a similar style or performing style conversion on other speech.

[0077] In some embodiments, the electronic device may perform style encoding on the first audio segment through a style encoder. Accordingly, the electronic device performs style encoding on the first audio segment to obtain a prompt style sequence, including: first, the electronic device normalizes the first audio segment. Then, the electronic device obtains the normalized Mel spectrogram of the first audio segment. Finally, the electronic device inputs the Mel spectrogram of the first audio segment into the style encoder to obtain a prompt style sequence. Style encoding of the first audio segment helps to accurately extract the speaking style of the object in the first audio segment, providing a reliable style basis for subsequent speech processing, so that speech with a specific style can be synthesized or style conversion can be performed.

[0078] Optionally, the structure of the style encoder is a convolutional neural network or a long short-term memory network. The input of the style encoder is an audio feature, such as a mel-spectrogram, and the output is a feature vector of the style, which is called a prompt style sequence in the embodiment of the present disclosure. The prompt style sequence can play the role of prompt information.

[0079] In some embodiments, the electronic device may also align the first audio segment and the voice text in a time dimension so that the length of the morphemes in the voice text is consistent with the length of the corresponding voice segment in the first audio segment. By aligning the first audio segment and the voice text in a time dimension, it is ensured that the length of the morphemes in the voice text is consistent with the length of the corresponding voice segment in the first audio segment, thereby facilitating a more accurate application of the speaking style of the first audio segment to the voice text, and providing a more accurate basis for the subsequent synthesis of speech with a specific style.

[0080] In step S303, the phoneme sequence and the prompt style sequence are input into the style language model to obtain a predicted style sequence.

[0081] In the disclosed embodiment, the style language model is used to capture the relationship between phonemes and styles through an attention mechanism. Since the phoneme sequence is converted from the speech text, it represents the basic pronunciation unit of speech. The prompt style sequence is obtained by style encoding the first audio clip, reflecting the speaking style of a specific object. Therefore, by inputting these two sequences into the style language model, the model can better understand the correspondence between phonemes and styles during the processing based on the attention mechanism. The predicted style sequence can be used in subsequent processing to synthesize speech with a specific style, so that the synthesized speech has both the content of the speech text and the speaking style of the first object.

[0082] In some embodiments, the style language model uses a Transformer or Seq2Seq architecture, takes a phoneme sequence and a style sequence as input, captures the relationship between the phonemes and the style through an attention mechanism, and outputs a predicted style sequence.

[0083] In step S304, the second audio segment is encoded to obtain a speech embedding representation of the second object.

[0084] In the disclosed embodiment, the second audio clip contains the sound information of the second object speaking, and the second audio clip can be used to obtain the timbre of the second object. Timbre refers to the characteristics of the speaker's voice, and different speakers have different timbres. The speech embedding representation is a digital abstract description of the timbre characteristics of the second object, which can be used in subsequent processing to combine with other elements (such as style, etc.) to synthesize speech with the timbre characteristics of the second object. Optionally, the speech embedding representation is a speech embedding vector.

[0085] In some embodiments, the electronic device encodes the second audio segment using a speaker encoder. Optionally, the speaker encoder is obtained by fine-tuning a pre-trained speech recognition model.

[0086] In step S305, the speech embedding representation and the predicted style sequence are input into the acoustic diffusion model to obtain a target mel-spectrogram.

[0087] In the disclosed embodiment, the acoustic diffusion model is used to gradually generate a mel-spectrogram from noise using a reverse diffusion process. Since the speech embedding representation is obtained after encoding the second audio clip, it represents the timbre characteristics of the second object; the predicted style sequence is obtained by inputting the phoneme sequence and the prompt style sequence into the style language model, which reflects the relationship between the speaking style of the first object and the phonemes of the speech text. The reverse diffusion process is a method of gradually removing noise to restore the original signal. Therefore, by feeding these two inputs into the acoustic diffusion model, a mel-spectrogram can be gradually generated from noise using a reverse diffusion process. Among them, the acoustic diffusion model gradually generates a target mel-spectrogram by combining the timbre characteristics of the second object (speech embedding representation) and the style of the first object (predicted style sequence). The target mel-spectrogram is a representation that can reflect the energy distribution of sound at different frequencies. It not only contains the timbre of the second object, but also reflects the speaking style of the first object, which provides an important basis for the subsequent synthesis of predicted speech that expresses the speech text content in the speaking style of the first object and the timbre of the second object.

[0088] In some embodiments, the acoustic diffusion model is a generative model based on a diffusion process, and the acoustic diffusion model can gradually generate a mel-spectrogram from noise using a reverse diffusion process.

[0089] In step S306, the target mel-spectrogram is input into the waveform diffusion model to generate predicted speech.

[0090] In the disclosed embodiment, the target mel-spectrogram contains both the speaking style of the first subject and the timbre of the second subject. The waveform diffusion model can generate a high-fidelity audio signal by gradually removing noise using the reverse diffusion process. The waveform diffusion model can start from the target mel-spectrogram, gradually remove noise, and finally generate a predicted speech. The predicted speech is a synthesized speech that expresses the given speech text content in the speaking style of the first subject and the timbre of the second subject, and has a high fidelity, that is, it can restore the natural sound effect as realistically as possible.

[0091] In some embodiments, the waveform diffusion model is constructed based on a diffusion model or other generative models to generate a high-fidelity audio signal by progressive denoising. Optionally, the acoustic diffusion model and waveform diffusion model can also be replaced by other types of generative models, such as a variational autoencoder (VAE) or a generative adversarial network (GAN) to replace the diffusion model.

[0092] It should be noted that the model involved in the embodiments of the present disclosure can be trained using a large-scale data set, and then an adaptive learning rate optimizer is used, and the model is optimized in combination with a loss function. The embodiments of the present disclosure do not limit the training process. Optionally, after the model training is completed, the model performance can be evaluated by subjective tests (such as MOS scores) and objective indicators (such as audio similarity). The embodiments of the present disclosure do not limit the model evaluation process.

[0093] It should be noted that in order to make the speech synthesis method provided by the embodiment of the present disclosure easier to understand, see Figure 4 shown. Figure 4 FIG. 1 is a flowchart of a speech synthesis method provided according to an exemplary embodiment. Figure 4 As shown, the speech text is input into the text-to-phoneme module to obtain a phoneme sequence. The first audio clip is input into the style encoder to obtain a prompt style sequence. The phoneme sequence and the prompt style sequence are input into the style language model to obtain a predicted style sequence. The second audio clip is input into the speaker encoder to obtain a speech embedding representation. The speech embedding representation and the predicted style sequence are input into the acoustic diffusion model to obtain a target mel-spectrogram. The target mel-spectrogram is input into the waveform diffusion model to generate predicted speech.

[0094] The disclosed embodiment provides a speech synthesis solution, which can be more natural and rich in style transfer based on the relationship between the speaking style of the first subject and the phonemes of the speech text by obtaining a predicted style sequence. By first obtaining the predicted style sequence, then obtaining the timbre, and then generating a mel-spectrogram, the style and timbre can be effectively decoupled, reducing the timbre leakage problem. Therefore, the predicted speech synthesized by the above-mentioned speech synthesis method has accurate timbre and good style expression.

[0095] Figure 5 FIG. 1 is a block diagram of a speech synthesis device according to an exemplary embodiment. Figure 5 As shown, the device includes: an acquisition unit 501, a first determination unit 502 and a second determination unit 503.

[0096] An acquisition unit 501 is configured to acquire a predicted style sequence based on the speech text and a first audio segment, wherein the first audio segment is used to acquire a speaking style of a first subject, and the predicted style sequence is used to represent a relationship between the speaking style of the first subject and the phonemes of the speech text;

[0097] A first determining unit 502 is configured to determine a target mel-spectrogram based on a second audio segment and a predicted style sequence, wherein the second audio segment is used to obtain a timbre of a second object;

[0098] The second determining unit 503 is configured to determine the predicted speech based on the target mel-spectrogram, where the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

[0099] In some embodiments, the acquisition unit 501 is configured to convert the speech text into a phoneme sequence; perform style encoding on the first audio clip to obtain a prompt style sequence; input the phoneme sequence and the prompt style sequence into a style language model to obtain a predicted style sequence, and the style language model is used to capture the relationship between phonemes and styles through an attention mechanism.

[0100] In some embodiments, the acquisition unit 501 is configured to normalize the first audio segment; acquire the normalized mel-spectrogram of the first audio segment; and input the mel-spectrogram of the first audio segment into a style encoder to obtain a prompt style sequence.

[0101] In some embodiments, the acquisition unit 501 is further configured to align the first audio segment and the speech text in a time dimension so that the length of the morphemes in the speech text is consistent with the length of the corresponding speech segment in the first audio segment.

[0102] In some embodiments, the first determination unit 502 is configured to encode the second audio segment to obtain a speech embedding representation of the second object; input the speech embedding representation and the predicted style sequence into the acoustic diffusion model to obtain a target mel-spectrogram, and the acoustic diffusion model is used to gradually generate a mel-spectrogram from noise using a reverse diffusion process.

[0103] In some embodiments, the second determining unit 503 is configured to input the target mel-spectrogram into a waveform diffusion model to generate predicted speech, and the waveform diffusion model is used to generate a high-fidelity audio signal by stepwise denoising using a reverse diffusion process.

[0104] The disclosed embodiment provides a speech synthesis device, which can be more natural and rich in style transfer based on the relationship between the speaking style of the first object and the phonemes of the speech text by obtaining a predicted style sequence. By first obtaining the predicted style sequence, then obtaining the timbre, and then generating a mel-spectrogram, the style and timbre can be effectively decoupled, reducing the timbre leakage problem. Therefore, the predicted speech synthesized by the above-mentioned speech synthesis method has accurate timbre and good style expression.

[0105] It should be noted that the speech synthesis device provided in the above embodiment is only illustrated by the division of the above functional units. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the speech synthesis device provided in the above embodiment and the speech synthesis method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0106] Regarding the speech synthesis device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0107] In the embodiments of the present application, the electronic device may be a terminal or a server. When the electronic device is a terminal, the terminal is used as the execution subject to implement the technical solution provided in the embodiments of the present application; when the electronic device is a server, the server is used as the execution subject to implement the technical solution provided in the embodiments of the present application; or, the technical solution provided in the present application is implemented through interaction between the terminal and the server. The embodiments of the present application are not limited to this.

[0108] Figure 6 6 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 600 includes: a processor 601 and a memory 602 .

[0109] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0110] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one program code, which is used to be executed by the processor 601 to implement the speech synthesis method provided by the method embodiment of the present disclosure.

[0111] In some embodiments, the electronic device 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607 and a power supply 608.

[0112] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0113] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0114] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the electronic device 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the electronic device 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0115] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0116] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0117] The power supply 608 is used to power various components in the electronic device 600. The power supply 608 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 608 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0118] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0119] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the above instructions can be executed by a processor 601 of an electronic device 600 to complete the training method of the above speech processing model, or to complete the above speech processing method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0120] A computer program product includes a computer program, which, when executed by a processor, implements the training method of the above-mentioned speech processing model, or implements the above-mentioned speech processing method.

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Based on the speech text and the first audio segment, obtaining a predicted style sequence, wherein the first audio segment is used to obtain a speaking style of a first subject, and the predicted style sequence is used to represent a relationship between the speaking style of the first subject and the phonemes of the speech text; Determining a target mel-spectrogram based on a second audio segment and the predicted style sequence, wherein the second audio segment is used to obtain the timbre of a second object; Based on the target mel-spectrogram, a predicted speech is determined, where the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

2. The speech synthesis method according to claim 1, characterized in that: The step of acquiring a predicted style sequence based on the speech text and the first audio segment includes: Converting the speech text into a phoneme sequence; Performing style encoding on the first audio clip to obtain a prompt style sequence; The phoneme sequence and the prompt style sequence are input into a style language model to obtain the predicted style sequence, and the style language model is used to capture the relationship between phonemes and styles through an attention mechanism.

3. The speech synthesis method according to claim 2, characterized in that: The performing style encoding on the first audio segment to obtain a prompt style sequence includes: Normalizing the first audio segment; Obtaining a normalized mel-spectrogram of the first audio segment; The mel-spectrogram of the first audio segment is input into a style encoder to obtain the prompt style sequence.

4. The speech synthesis method according to claim 1, characterized in that: The method further comprises: The first audio segment and the speech text are aligned in a time dimension so that the length of the morphemes in the speech text is consistent with the length of the corresponding speech segment in the first audio segment.

5. The speech synthesis method according to claim 1, characterized in that: The step of determining a target mel-spectrogram based on the second audio segment and the predicted style sequence includes: Encoding the second audio segment to obtain a speech embedding representation of the second object; The speech embedding representation and the predicted style sequence are input into an acoustic diffusion model to obtain the target mel-spectrogram, and the acoustic diffusion model is used to gradually generate a mel-spectrogram from noise using a reverse diffusion process.

6. The speech synthesis method according to claim 1, characterized in that: The step of determining the predicted speech based on the target Mel-spectrogram comprises: The target mel-spectrogram is input into a waveform diffusion model to generate the predicted speech, and the waveform diffusion model is used to generate a high-fidelity audio signal by gradually denoising using a reverse diffusion process.

7. A speech synthesis device, characterized in that: The device comprises: an acquisition unit configured to acquire a predicted style sequence based on a speech text and a first audio segment, wherein the first audio segment is used to acquire a speaking style of a first subject, and the predicted style sequence is used to represent a relationship between the speaking style of the first subject and phonemes of the speech text; A first determining unit is configured to determine a target mel-spectrogram based on a second audio segment and the predicted style sequence, wherein the second audio segment is used to obtain a timbre of a second object; The second determination unit is configured to determine a predicted speech based on the target mel-spectrogram, where the predicted speech is a synthesized speech that expresses the content of the speech text in the speaking style of the first subject and the timbre of the second subject.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the speech synthesis method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the speech synthesis method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images