Speech synthesis method and device, live broadcast system, electronic equipment and storage medium

Abstract

The invention relates to a voice synthesis method and device, a live broadcast system, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a target text of language synthesis and timbre characteristics of a speaker; coding the target guide audio to obtain a target discrete semantic code without timbre; wherein the target discrete semantic codes comprise rhythm features; predicting the target discrete semantic code and the target text based on a large language model to obtain a to-be-synthesized audio code; wherein the to-be-synthesized audio code comprises a rhythm of a target guide audio and a semantic feature of a target text; decoding the to-be-synthesized audio code according to the target text and the timbre feature to obtain a speech synthesis audio; according to the technical scheme, the synthesized voice can be close to a real person voice effect, the rhythm and tone of the synthesized voice can be modulated, and diversified application requirements in network live broadcast are met.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, device, live broadcast system, electronic device and computer-readable storage medium. Background Art

[0002] At present, with the rapid development of artificial intelligence and machine learning technologies, TTS (Text To Speech, speech synthesis technology) has gradually become an important research direction in the field of human-computer interaction; TTS can convert text information into natural and fluent speech output and is widely used in smart assistants, navigation systems, education, entertainment and other fields.

[0003] With the development of network technology, live streaming has been used by most network users. With its intuitive, immediate and interactive content and form, live streaming has played an important role in promoting flexible employment, promoting economic and social development, and enriching the spiritual and cultural life of the people. The anchors can better show their talents in live streaming, thus realizing the self-worth of more anchors. In live streaming platforms, TTS has been increasingly used. Conventional TTS technology is mostly mechanical. In recent years, TTS technology based on LLM (Large Language Models) has become the mainstream trend due to its high naturalness. For example, there is a technology that uses an audio encoder to encode continuous speech features into audio tokens (i.e. discrete semantic encoding), uses a large language model to establish the relationship between text and audio tokens, and then uses a vocoder to restore speech from audio tokens.

[0004] However, in the field of live streaming technology, although the existing LLM-based TTS technology has achieved good results in speech synthesis, it still has many limitations in terms of tone modulation, rhythm changes, etc. Practical applications often rely heavily on the sound quality of the recorded sound and specific template files, and it is difficult to meet the diverse application needs in live streaming. Summary of the invention

[0005] Based on this, it is necessary to provide a speech synthesis method, device, live broadcast system, electronic device and computer-readable storage medium that can generate high-quality live broadcast room summary information.

[0006] A speech synthesis method, comprising:

[0007] Obtaining the target text for speech synthesis and the speaker's timbre characteristics;

[0008] Encoding the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature;

[0009] Predicting the target discrete semantic code and the target text based on a large language model to obtain an audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target guide audio and the semantic features of the target text;

[0010] The audio code to be synthesized is decoded according to the target text and timbre characteristics to obtain speech synthesis audio.

[0011] In one embodiment, encoding the target guided audio to obtain a target discrete semantic code with timbre removed includes:

[0012] The target-guided audio input is based on the audio feature extraction model decoupled from the speaker's timbre to obtain semantic features that remove timbre and contain prosodic information;

[0013] The semantic features of the target guided audio are input into the residual quantization model to obtain the target discrete semantic encoding.

[0014] In one embodiment, predicting the target discrete semantic code and the target text based on a large language model to obtain the audio code to be synthesized includes:

[0015] Calling a pre-trained large language model; wherein the large language model is trained based on a general language model and using training text and training audio;

[0016] The target discrete semantic code and the target text are input into the large language model for prediction to obtain the audio code to be synthesized including the rhythm of the target guided audio and the semantic features of the target text.

[0017] In one embodiment, the speech synthesis method further includes:

[0018] Build large language models;

[0019] Obtain a text-audio pair dataset including training audio and its corresponding text content;

[0020] Input the training audio into the encoder to obtain a discrete semantic code of the training audio; wherein the discrete semantic code is the target of large language model optimization;

[0021] Input the discrete semantic code of the training audio and the text content into the large language model to predict and obtain the discrete semantic code of the audio to be synthesized;

[0022] Based on the discrete semantic coding of the audio to be synthesized and the discrete semantic coding of the training audio, the large language model is optimized using cross entropy as the objective function.

[0023] In one embodiment, decoding the audio code to be synthesized according to the target text and the timbre feature to obtain speech synthesis audio includes:

[0024] Calling a pre-trained decoder; wherein the decoder is trained based on training text and the speaker's timbre characteristics;

[0025] The audio code to be synthesized, the target text and the timbre characteristics are input into the decoder for decoding to obtain speech synthesis audio.

[0026] In one embodiment, the speech synthesis method further includes:

[0027] Constructing a decoder; wherein the decoder comprises: a priori encoder, a posteriori encoder, a regular stream, a generator and a discriminator;

[0028] Setting a loss function; wherein the loss function includes: L1 reconstruction loss, KL divergence, adversarial loss and feature matching loss;

[0029] The training text and the training audio are input into the decoder, and the decoder is jointly optimized based on the loss function.

[0030] In one embodiment, the KL divergence is used to measure the difference between the posterior distribution and the prior distribution;

[0031] The L1 reconstruction loss is used to measure the difference between the spectrogram predicted by the generator and the target spectrogram;

[0032] The adversarial loss is used to train the generator and the discriminator;

[0033] The feature matching loss is used to optimize the generator.

[0034] In one embodiment, the audio code to be synthesized, the target text and the timbre feature are input into a pre-trained decoder for decoding to obtain speech synthesis audio, including:

[0035] The audio code to be synthesized, the target text and the timbre features are passed through a priori encoder to obtain a priori distribution, the priori distribution is passed through a regular stream to obtain a posterior distribution, and the posterior distribution and the timbre features are input into a generator for fusion to obtain speech synthesis audio.

[0036] A speech synthesis device, comprising:

[0037] An input module, used to obtain the target text for speech synthesis and the speaker's timbre characteristics;

[0038] An encoding module, used for encoding the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature;

[0039] A prediction module, configured to predict the target discrete semantic code and the target text based on a large language model to obtain an audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target guide audio and the semantic features of the target text;

[0040] A decoding module is used to decode the audio code to be synthesized according to the target text and timbre characteristics to obtain speech synthesis audio.

[0041] A live broadcast system, comprising: an anchor terminal, an audience terminal and a live broadcast server; wherein the anchor terminal and the audience terminal are respectively connected to the live broadcast server via a communication network;

[0042] The anchor terminal is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server;

[0043] The live broadcast server is used to forward the live broadcast video stream of the anchor to the audience end, and use the speech synthesis method to generate the audio stream in the live broadcast video stream;

[0044] The audience end is used for audience users who access the live broadcast room, receive the live video stream of the anchor for playback, and display the push information.

[0045] An electronic device, comprising:

[0046] one or more processors;

[0047] Memory;

[0048] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the steps of the speech synthesis method.

[0049] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes the steps of the speech synthesis method.

[0050] The technical solution of the present application obtains discrete semantic codes of the guide audio through encoding, obtains the audio codes to be synthesized through the large language model for predicting the discrete semantic codes and the target text, and obtains the speech synthesis audio by decoding the audio codes to be synthesized in combination with the timbre characteristics of the speaker; with this technical solution, the synthesized speech can be close to the effect of real human speech, and the rhythm and timbre of the synthesized speech can be adjusted based on the large language model, so as to create a unique timbre and rhythm, improve the speech synthesis effect, and meet the diversified application needs in live broadcasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of an example live broadcast service application scenario;

[0052] Figure 2 This is a flowchart of an example speech synthesis algorithm;

[0053] Figure 3 is a flow chart of a speech synthesis method according to an embodiment;

[0054] Figure 4 is a schematic diagram of training a large language model of an embodiment;

[0055] Figure 5 is a schematic diagram of a decoder training process of an embodiment;

[0056] Figure 6 is a streaming flow chart of a decoder of an embodiment;

[0057] Figure 7 is a schematic diagram of the structure of a speech synthesis device according to an embodiment;

[0058] Figure 8 This is a schematic diagram of the live broadcast system structure;

[0059] Fig. 9 is a block diagram of an example electronic device. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. For example, multiple objects refer to two or more objects. "Include" or "comprising" and other similar words mean that the information appearing before "include" or "comprising" covers the information listed after "include" or "comprising" and its equivalents, and does not exclude other information. The "and / or" mentioned in the embodiments of the present application indicates that there can be three kinds of relationships. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0062] The technical solution provided in the embodiments of the present application can be applied to Figure 1 In the application scenario of the method related to this application shown in FIG. Figure 1This is a schematic diagram of an example of a live broadcast service application scenario. The live broadcast system may include a live broadcast server, an anchor terminal and an audience terminal. The anchor terminal and the audience terminal communicate with the live broadcast server through a communication network, so that the anchor at the anchor terminal and the audience users at the audience terminal can perform real-time network live broadcast. Among them, for the anchor terminal and the audience terminal, their terminal devices can be but are not limited to various personal computers, laptops, smart phones and tablets. The live broadcast server can be implemented by an independent server or a server cluster composed of multiple servers.

[0063] The technical solution of the embodiment of this application refers to Figure 2 As shown, Figure 2 This is an example flowchart of a speech synthesis algorithm, which uses the guide audio and the input language to synthesize the target text and the speaker's timbre features, passes the guide audio through an encoder to obtain a discrete semantic code containing rhythm after removing the timbre, passes the discrete semantic code and the target text through a trained large language model to obtain a discrete semantic code of the audio to be synthesized containing the rhythm of the guide audio and the semantic features of the target text; passes the discrete semantic code of the audio to be synthesized, the target text and the speaker's timbre features through a trained decoder to obtain the final speech synthesis audio; this technical solution can be used in AI-assisted voice live broadcast scenarios, not only can the synthesized speech be close to the real person effect, but the rhythm and timbre of the synthesized speech can also be modulated at will, and by adjusting the technology of guiding audio and timbre mixing, a unique timbre and rhythm can be created to enhance the interactivity of the live broadcast.

[0064] refer to Figure 3 As shown, Figure 3 The flowchart of the speech synthesis method of an embodiment includes the following steps:

[0065] Step S30, obtaining the target text for speech synthesis and the speaker's timbre characteristics.

[0066] In this step, the target text for speech synthesis is generated according to the text content, and the timbre characteristics of the speaker are produced; the timbre characteristics can be the timbre characteristics of a certain speaker, or the timbre characteristics of multiple speakers can be mixed; as in the above steps, the timbre characteristics can be set according to the needs to modulate the corresponding rhythm and timbre characteristics.

[0067] Step S20, encoding the target guide audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature.

[0068] In this step, the target guide audio is encoded by an encoder to remove the timbre, thereby obtaining a target discrete semantic code including rhythm; wherein the guide audio can be a speech audio for a speaker to read after, or any pure human voice of a speaker.

[0069] In one embodiment, step S20 encodes the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code contains prosodic features, including:

[0070] The target-guided audio is input into an audio feature extraction model based on speaker timbre decoupling to obtain semantic features with timbre removed and containing prosodic information; the semantic features of the target-guided audio are input into a residual quantization model for processing to obtain target discrete semantic coding.

[0071] Specifically, the encoder may include a pre-trained audio feature extraction model (ContentVec) and a residual vector quantization model (RVQ) for speaker timbre decoupling. The target guided audio is subjected to the audio feature extraction model to obtain semantic features without timbre, which mainly include prosodic information. The semantic features of the guided audio are subjected to the residual vector quantization model to obtain corresponding discrete semantic codes (tokens). Tokens are the input information of the large language model (LLM), which is called discrete semantic codes.

[0072] Step S30, predicting the target discrete semantic code and the target text based on the large language model to obtain the audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target leading audio and the semantic features of the target text.

[0073] In this step, the tokens of the target guide audio and the target text obtained by encoding are used to predict the tokens of the audio to be synthesized including the rhythm of the target guide audio and the semantic features of the target text through the pre-trained LLM model.

[0074] In one embodiment, the step S30 of predicting the target discrete semantic code and the target text based on the large language model to obtain the audio code to be synthesized includes:

[0075] (1) Calling a pre-trained large language model; wherein the large language model is trained based on a general language model using training text and training audio.

[0076] As an example, refer to Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram of training a large language model of an embodiment. A training method for a large language model may include the following:

[0077] a. Build a large language model;

[0078] b. Obtain a text-audio pair dataset including training audio and its corresponding text content; specifically, through a massive text-audio pair dataset Among them, a i is the training audio, x i It is the text content corresponding to the training audio.

[0079] c. Input the training audio into an encoder to obtain a discrete semantic encoding of the training audio; wherein the discrete semantic encoding is the goal of large language model optimization.

[0080] d. Input the discrete semantic code of the training audio and the text content into the large language model to predict the discrete semantic code of the audio to be synthesized; specifically, the training audio a i First, the encoder is used to obtain the corresponding semantic code y to be discretized i , discrete semantic encoding y i In addition to being an optimization target for LLM learning, it is also related to the text content x i Enter LLM together to obtain the predicted discrete semantic code y′ of the audio to be synthesized i .

[0081] e. Based on the discrete semantic coding of the audio to be synthesized and the discrete semantic coding of the training audio, the large language model is optimized using cross entropy as the objective function.

[0082] Specifically, the cross entropy loss calculation formula is as follows:

[0083]

[0084] In the formula, Loss(y′ i ,y i ) is the cross entropy loss.

[0085] (2) The target discrete semantic code and the target text are input into the large language model for prediction to obtain the audio code to be synthesized including the rhythm of the target guided audio and the semantic features of the target text.

[0086] Specifically, the tokens of the guide audio and the target text obtained in step S10 are input into the trained LLM to obtain the predicted tokens of the audio to be synthesized.

[0087] Step S40: decoding the audio code to be synthesized according to the target text and the timbre characteristics to obtain speech synthesis audio.

[0088] In this step, the target text, timbre features and the audio code to be synthesized are decoded by a pre-trained decoder to obtain the final speech synthesis audio output.

[0089] In one embodiment, when step S40 obtains speech synthesis audio through decoding, a pre-trained decoder is called, and the audio code to be synthesized, the target text and the timbre characteristics are input into the decoder for decoding to obtain speech synthesis audio; wherein the decoder is trained based on the training text and the timbre characteristics of the speaker.

[0090] In one embodiment, reference Figure 5 As shown, Figure 5 The following is a schematic diagram of a decoder training process according to an embodiment. The training method of the decoder may be as follows:

[0091] 1) Constructing a decoder; wherein the decoder includes a priori encoder, a posterior encoder, a regular stream, a generator and a discriminator model.

[0092] like Figure 5 As shown, a decoder model including a priori encoder, a posteriori encoder, a normalizing flow, a generator, and a discriminator model can be constructed.

[0093] 2) Setting a loss function; wherein the loss function includes: L1 reconstruction loss, KL divergence, adversarial loss and feature matching loss.

[0094] Specifically, by inputting training text and training audio, four loss functions, namely L1 reconstruction loss, KL divergence, adversarial loss and feature matching loss, are used to jointly optimize the decoder.

[0095] For L1 reconstruction loss, the target spectrogram x is obtained for training audio. mel , the target spectrum is passed through the posterior encoder to obtain the hidden variable z, and the hidden variable is passed through the generator to obtain the predicted spectrum graph Based on the target spectrogram and the predicted spectrogram, L1 reconstruction loss is finally used for optimization.

[0096] KL divergence loss, KL divergence is the distance between the difference between one probability distribution and another probability distribution. It is used to train audio to pass through the encoder to obtain its discrete semantic encoding, and then pass through the prior encoder together with the training text and the speaker's timbre to obtain the prior distribution. The latent variable z passes through Flow to obtain the posterior distribution, and KL divergence loss is used for optimization based on the prior distribution and the posterior distribution.

[0097] The adversarial loss is to train the generator and discriminator so that the predicted spectrogram More real

[0098] Feature matching loss is used to optimize the generator, and the expected input target spectrum map x mel and predicted spectrogram When , the feature maps of each layer of the generator can be similar.

[0099] 3) Inputting the training text and the training audio into the decoder, and jointly optimizing the decoder based on the loss function.

[0100] Specifically, the prior encoder is a variational inference (VAE) structure that uses normalizing flows to enhance the expressiveness of the prior distribution. Normalizing flows are used to learn complex data distributions. Generative Adversarial Networks (GAN) training is used to improve the quality of synthesized audio. The final loss function combines VAE and GAN training, including reconstruction loss, KL divergence, adversarial loss, and feature matching loss.

[0101] Exemplarily, the reconstruction loss uses L1 loss to measure the difference between the generator's predicted spectrogram and the target spectrogram, and the formula is as follows:

[0102]

[0103] Among them, x mel is the target spectrum, is the predicted spectrum graph; the difference between the posterior distribution and the prior distribution is measured by the KL divergence, and the formula is as follows:

[0104] L kl =log(q φ )-log(p θ )

[0105] Among them, q φ is the posterior distribution, p θ is the prior distribution.

[0106] The adversarial loss includes the loss of the discriminator and the loss of the generator, where the loss formula of the discriminator is as follows:

[0107] L adv (D)=E (y,a) [(D(a)-1) 2 +(D(G(z))) 2 ]

[0108] The loss formula of the generator is as follows:

[0109] Ladv(G)=E z [(D(G(z))-1) 2 ]

[0110] In the formula, D is the discriminator, G is the generator, a is the input audio, and z is the latent variable output by the posterior encoder; the feature matching loss is mainly used to train the generator, and the formula is as follows:

[0111]

[0112] Among them, D l is the feature map output by the discriminator’s layer l, N l is the number of feature maps.

[0113] Combining all the above loss functions to get the total loss function, the formula is as follows:

[0114] L total =L recon +L kl +L adv (G)+L fm (G)

[0115] Among them, L total Represents the total loss function.

[0116] In one embodiment, when a decoder is used to decode and obtain speech synthesis audio, the audio code to be synthesized, the target text and the timbre features are passed through a priori encoder to obtain a priori distribution, the priori distribution is passed through a normal stream to obtain a posterior distribution, and the posterior distribution and the timbre features are input into a generator for fusion to obtain speech synthesis audio.

[0117] Specifically, based on the above training, the decoder is obtained, refer to Figure 6 As shown, Figure 6 This is a streaming flowchart of a decoder of an embodiment. When performing inference, the target text, tokens predicted by LLM, and speaker timbre characteristics are received to finally obtain speech synthesis audio. The rhythm of the speech synthesis audio depends on the guide audio, and the timbre depends on the speaker's timbre characteristics.

[0118] Preferably, the speaker's timbre feature can specify multiple timbres for fusion. For example, assuming there are n timbre features, represented as e1, e2, ..., e n , the corresponding fusion ratio is p1,p2,…,p n , and satisfies The formula for timbre fusion is: embedding = e1×p1+e2×p2+…+e n ×p n , thereby meeting the needs for timbre diversity and reducing the use of timbre characteristics of specific speakers.

[0119] The technical solution of the above-mentioned embodiment can be used for AI assistants in live broadcast rooms. By adding AI live broadcast auxiliary functions, users can customize the voice lines of AI assistants independently, so that the synthesized voice can have a rhythm close to that of real people. For example, in the group chat scene of the live broadcast platform, the function of group chat between multiple virtual people and users can be realized, and the voice of the host and the non-host can be virtualized. The voice effect synthesized by the host's voice can be close to that of a real person, and the non-host's voice can be modulated to meet the expected voice and rhythm. By adjusting the technology of guiding audio and voice mixing, a unique voice and rhythm can be created to meet various business needs.

[0120] An embodiment of a speech synthesis device is described below.

[0121] like Figure 7 As shown, Figure 7 The structure diagram of a speech synthesis device of an embodiment includes:

[0122] An input module 10 is used to obtain the target text for speech synthesis and the timbre characteristics of the speaker;

[0123] The encoding module 20 is used to encode the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature;

[0124] A prediction module 30 is used to predict the target discrete semantic code and the target text based on a large language model to obtain an audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target guide audio and the semantic features of the target text;

[0125] The decoding module 40 is used to decode the audio code to be synthesized according to the target text and timbre characteristics to obtain speech synthesis audio.

[0126] The speech synthesis device of this embodiment can execute a speech synthesis method provided in the embodiment of the present application. The implementation principle is similar. The actions performed by each module in the speech synthesis device in each embodiment of the present application correspond to the steps in the speech synthesis method in each embodiment of the present application. For the detailed functional description of each module of the speech synthesis device, please refer to the description in the corresponding speech synthesis method shown in the previous text, which will not be repeated here.

[0127] An embodiment of the live broadcast system is described below.

[0128] The live broadcast system of this embodiment refers to Figure 8 As shown, Figure 8This is an example of a live broadcast system structure diagram. The system framework may include an anchor terminal, a live broadcast server and an audience terminal. The audience terminal and the anchor terminal may access the live broadcast platform through a client installed on an electronic device. For example, the anchor terminal and the audience terminal may be computer devices such as PDAs, smart phones, tablet computers, desktop computers or laptop computers, etc., but this is not limited. They may also be software modules of an application program. The live broadcast server may be implemented with an independent server or a server cluster composed of multiple servers. The anchor terminal is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server; the live broadcast server is used to forward the anchor's live video stream to the audience terminal, and use the speech synthesis method of any of the aforementioned embodiments to generate the audio stream in the live video stream; the audience terminal is used to access the audience users in the live broadcast room and receive the anchor's live video stream for playback.

[0129] For example, an AI assistant is used in a live broadcast room to perform voice broadcast. Users can customize the voice line of the AI ​​assistant and synthesize the live broadcast voice so that the synthesized voice has a rhythm close to that of a real person. For example, the function of group chat between multiple virtual people and users can be realized, and the voice of the host and the voice of non-host can be virtualized. The AI ​​assistant can create a unique voice and rhythm to meet various business needs.

[0130] The technical solution of the present application can synthesize speech that is close to the effect of real human speech, and can adjust the rhythm and timbre based on a large language model, so as to modulate the rhythm and timbre of the synthesized speech, thereby creating a unique timbre and rhythm, improving the speech synthesis effect, and meeting the diverse application needs in live streaming.

[0131] Embodiments of electronic devices and computer-readable storage media are described below.

[0132] The present application provides a technical solution for an electronic device to implement functions related to a speech synthesis method. The electronic device of this embodiment includes one or more processors, a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured for the steps of the speech synthesis method of any embodiment.

[0133] like Fig. 9 As shown, Fig. 9 1 is a block diagram of an exemplary electronic device; the electronic device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc. The electronic device 100 may include one or more of the following components: a processing component 102, a memory 104, a power component 106, a multimedia component 108, an audio component 109, an input / output (I / O) interface 112, a sensor component 114, and a communication component 116.

[0134] The processing component 102 generally controls the overall operation of the electronic device 100, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations.

[0135] The memory 104 is configured to store various types of data to support operations in the electronic device 100, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0136] The power supply component 106 provides power to various components of the electronic device 100 .

[0137] The multimedia component 109 includes a screen that provides an output interface between the electronic device 100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). In some embodiments, the multimedia component 108 includes a front camera and / or a rear camera.

[0138] The audio component 109 is configured to output and / or input audio signals.

[0139] I / O interface 112 provides an interface between processing component 102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0140] The sensor component 114 includes one or more sensors for providing various aspects of status assessment for the electronic device 100. The sensor component 114 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact.

[0141] The communication component 116 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices. The electronic device 100 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof.

[0142] The present application provides a technical solution of a computer-readable storage medium for realizing functions related to a speech synthesis method. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes the speech synthesis method of any embodiment.

[0143] In an exemplary embodiment, the computer-readable storage medium may be a non-temporary computer-readable storage medium including instructions, such as a memory including instructions. For example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0144] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining the target text for speech synthesis and the speaker's timbre characteristics; Encoding the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature; Predicting the target discrete semantic code and the target text based on a large language model to obtain an audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target guide audio and the semantic features of the target text; The audio code to be synthesized is decoded according to the target text and timbre characteristics to obtain speech synthesis audio.

2. The speech synthesis method according to claim 1, characterized in that: The target guided audio is encoded to obtain the target discrete semantic encoding without timbre, including: The target-guided audio input is based on the audio feature extraction model decoupled from the speaker's timbre to obtain semantic features that remove timbre and contain prosodic information; The semantic features of the target guided audio are input into the residual quantization model to obtain the target discrete semantic encoding.

3. The speech synthesis method according to claim 1, characterized in that: Predicting the target discrete semantic code and the target text based on the large language model to obtain the audio code to be synthesized includes: Calling a pre-trained large language model; wherein the large language model is trained based on a general language model and using training text and training audio; The target discrete semantic code and the target text are input into the large language model for prediction to obtain the audio code to be synthesized including the rhythm of the target guided audio and the semantic features of the target text.

4. The speech synthesis method according to claim 3, characterized in that: Also includes: Build large language models; Obtain a text-audio pair dataset including training audio and its corresponding text content; Input the training audio into the encoder to obtain a discrete semantic code of the training audio; wherein the discrete semantic code is the target of large language model optimization; Input the discrete semantic code of the training audio and the text content into the large language model to predict and obtain the discrete semantic code of the audio to be synthesized; Based on the discrete semantic coding of the audio to be synthesized and the discrete semantic coding of the training audio, the large language model is optimized using cross entropy as the objective function.

5. The speech synthesis method according to claim 1, characterized in that: Decoding the audio code to be synthesized according to the target text and the timbre characteristics to obtain speech synthesis audio includes: Calling a pre-trained decoder; wherein the decoder is trained based on training text and the speaker's timbre characteristics; The audio code to be synthesized, the target text and the timbre characteristics are input into the decoder for decoding to obtain speech synthesis audio.

6. The speech synthesis method according to claim 5, characterized in that: Also includes: Constructing a decoder; wherein the decoder comprises: a priori encoder, a posteriori encoder, a regular stream, a generator and a discriminator; Setting a loss function; wherein the loss function includes: L1 reconstruction loss, KL divergence, adversarial loss and feature matching loss; The training text and the training audio are input into the decoder, and the decoder is jointly optimized based on the loss function.

7. The speech synthesis method according to claim 6, characterized in that: The KL divergence is used to measure the difference between the posterior distribution and the prior distribution; The L1 reconstruction loss is used to measure the difference between the spectrogram predicted by the generator and the target spectrogram; The adversarial loss is used to train the generator and the discriminator; The feature matching loss is used to optimize the generator.

8. The speech synthesis method according to claim 6, characterized in that: Inputting the audio code to be synthesized, the target text and the timbre feature into a pre-trained decoder for decoding to obtain speech synthesis audio, including: The audio code to be synthesized, the target text and the timbre features are passed through a priori encoder to obtain a priori distribution, the priori distribution is passed through a regular stream to obtain a posterior distribution, and the posterior distribution and the timbre features are input into a generator for fusion to obtain speech synthesis audio.

9. A speech synthesis device, characterized in that: include: An input module, used to obtain the target text for speech synthesis and the speaker's timbre characteristics; An encoding module, used for encoding the target guided audio to obtain a target discrete semantic code with timbre removed; wherein the target discrete semantic code includes a rhythmic feature; A prediction module, configured to predict the target discrete semantic code and the target text based on a large language model to obtain an audio code to be synthesized; wherein the audio code to be synthesized includes the rhythm of the target guide audio and the semantic features of the target text; A decoding module is used to decode the audio code to be synthesized according to the target text and timbre characteristics to obtain speech synthesis audio.

10. A live broadcast system, characterized in that: include: An anchor terminal, an audience terminal and a live broadcast server; wherein the anchor terminal and the audience terminal are respectively connected to the live broadcast server via a communication network; The anchor terminal is used to access the anchor in the live broadcast room and collect the anchor's live video stream and upload it to the live broadcast server; The live broadcast server is used to forward the live broadcast video stream of the anchor to the audience end, and use the speech synthesis method described in any one of claims 1 to 8 to generate an audio stream in the live broadcast video stream; The audience end is used for audience users who access the live broadcast room, receive the live video stream of the anchor for playback, and display the push information.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the steps of the speech synthesis method described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes the steps of the speech synthesis method described in any one of claims 1-8.

Images