Voice generation device and training method thereof, electronic equipment and storage medium

Abstract

The invention relates to a voice generation device and a training method thereof, electronic equipment and a storage medium, and the device comprises a spoken text rewriting module which receives input prompt information and rewritten text, rewrites the rewritten text into a spoken text corresponding to the prompt information according to the prompt information, and outputs the spoken text corresponding to the prompt information; the prompt information is used for controlling the spoken language degree of the rewritten spoken language text; and the voice generation module is used for generating target voice corresponding to the spoken text based on the prompt information. According to the embodiment of the invention, the naturalness and anthropomorphic degree of voice generation are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech generation device and a training method thereof, an electronic device and a storage medium. Background Art

[0002] As an important branch of artificial intelligence, speech generation technology plays a key role in scenarios such as voice interaction and text reading. Although traditional speech generation methods have made significant progress in the naturalness and clarity of speech, they still have limitations in emotional expression and colloquialism.

[0003] In order to improve the naturalness of spoken speech generation, existing technologies use two methods. One is that the model automatically analyzes the text and generates audio with corresponding emotions and tones, but this method has low controllability and is difficult to meet users' needs for different tones. The other method is to use text rewriting technology to make the generated content more colloquial. Although the controllability is improved, the generated audio still cannot match the user's needs, resulting in a stiff and dull generation effect. Summary of the invention

[0004] The present disclosure proposes a speech generation technology solution.

[0005] According to one aspect of the present disclosure, there is provided a speech generating device, comprising:

[0006] A spoken text rewriting module receives inputted prompt information and a rewritten text, and rewrites the rewritten text into a spoken text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the spoken degree of the rewritten spoken text;

[0007] The speech generation module generates a target speech corresponding to the spoken text based on the prompt information.

[0008] In a possible implementation, the spoken text rewriting module includes a first large language model;

[0009] The first language model rewrites the rewritten text according to the scene information and / or character information in the prompt information to obtain a spoken text.

[0010] In a possible implementation, the spoken text contains emotional information and / or spoken events, and the emotional information and / or spoken events are represented based on emojis.

[0011] In a possible implementation, the speech generation module includes:

[0012] A voiceprint encoder, which encodes the reference audio corresponding to the personality information according to the personality information in the prompt information to obtain a voiceprint vector for representing the timbre of the personality information;

[0013] The second language model generates an audio code corresponding to the spoken text based on the voiceprint vector.

[0014] In a possible implementation, the speech generation module includes:

[0015] The audio decoder encodes the audio output by the second largest language model and decodes it into the target speech.

[0016] According to one aspect of the present disclosure, a training method for a speech generating device is provided, for training the above-mentioned device, the method comprising:

[0017] Determine the spoken text corresponding to the sample audio;

[0018] Encoding the sample audio to obtain audio code;

[0019] A speech generation training set is constructed based on the spoken text and the audio code, and a second largest language model in the speech generation module is trained, wherein the spoken text serves as an input of the second largest language model, and the audio code serves as an expected output of the second largest language model.

[0020] In a possible implementation, determining the spoken text corresponding to the sample audio includes:

[0021] Performing speech recognition on the sample audio to obtain recognized text;

[0022] Performing spoken event detection on the sample audio to obtain spoken events;

[0023] Performing emotion classification on the sample audio to obtain emotion information;

[0024] The emotional information and the spoken events are inserted into the recognized text to obtain the spoken text.

[0025] In a possible implementation manner, encoding the sample audio to obtain audio encoding includes:

[0026] Encode the input audio to obtain an audio vector;

[0027] A vector quantization operation is performed on the audio vector to obtain discrete audio code.

[0028] In a possible implementation, the speech generation training set also includes reference audio, the timbre of the reference audio is the same as the timbre of the sample audio, and the reference audio is used to control the timbre of the audio generated by the speech generation module.

[0029] In a possible implementation, the method further includes:

[0030] Based on the colloquial rewritten text, the text before rewriting and the prompt words, the colloquial text rewriting module is trained, wherein the text before rewriting and the prompt words serve as inputs of the colloquial text rewriting module, and the colloquial rewritten text serves as the expected output of the colloquial text rewriting module.

[0031] According to one aspect of the present disclosure, there is provided a speech generation method, comprising:

[0032] receiving inputted prompt information and rewritten text, and rewriting the rewritten text into a colloquial text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the colloquial degree of the rewritten colloquial text;

[0033] Based on the prompt information, a target speech corresponding to the colloquial text is generated.

[0034] In a possible implementation, the spoken text rewriting module includes a first large language model;

[0035] The first language model rewrites the rewritten text according to the scene information and / or character information in the prompt information to obtain a spoken text.

[0036] In a possible implementation, the spoken text contains emotional information and / or spoken events, and the emotional information and / or spoken events are represented based on emojis.

[0037] In a possible implementation, generating a target speech corresponding to the colloquial text based on the prompt information includes:

[0038] According to the character information in the prompt information, encoding the reference audio corresponding to the character information to obtain a voiceprint vector for representing the timbre of the character information;

[0039] The second largest language model is used to generate an audio code corresponding to the spoken text based on the voiceprint vector.

[0040] In a possible implementation, generating a target speech corresponding to the colloquial text based on the prompt information includes:

[0041] Encode the audio output of the second largest language model and decode it into the target speech.

[0042] According to one aspect of the present disclosure, a training device for a speech generating device is provided, for training the above-mentioned speech generating device, the training device comprising:

[0043] A spoken text determination module, used to determine the spoken text corresponding to the sample audio;

[0044] An encoding module, used for encoding the sample audio to obtain audio code;

[0045] The first training module is used to construct a speech generation training set based on the spoken text and the audio code, and train the second largest language model in the speech generation module, wherein the spoken text serves as the input of the second largest language model, and the audio code serves as the expected output of the second largest language model.

[0046] In a possible implementation, the spoken text determination module is used to:

[0047] Performing speech recognition on the sample audio to obtain recognized text;

[0048] Performing spoken event detection on the sample audio to obtain spoken events;

[0049] Performing emotion classification on the sample audio to obtain emotion information;

[0050] The emotional information and the spoken events are inserted into the recognized text to obtain the spoken text.

[0051] In a possible implementation, the encoding module is used to:

[0052] Encode the input audio to obtain an audio vector;

[0053] A vector quantization operation is performed on the audio vector to obtain discrete audio code.

[0054] In a possible implementation, the speech generation training set also includes reference audio, the timbre of the reference audio is the same as the timbre of the sample audio, and the reference audio is used to control the timbre of the audio generated by the speech generation module.

[0055] In a possible implementation, the training device further includes:

[0056] The second training module is used to train the spoken text rewriting module based on the spoken rewritten text, the text before rewriting and the prompt words, wherein the text before rewriting and the prompt words serve as inputs of the spoken text rewriting module, and the spoken rewritten text serves as the expected output of the spoken text rewriting module.

[0057] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0058] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0059] In the disclosed embodiment, through the rewriting of colloquial text and the generation of voice based on prompt information, the generated voice is closer to real conversation, the emotional expression is richer, and the naturalness and anthropomorphism of the voice generation are improved. In addition, through the control of prompt information (including scene information and / or character information), the user can flexibly adjust the colloquial degree, emotional expression, etc. of the generated voice, enhance the controllability of voice generation, so as to be suitable for different voice interaction scenarios, such as smart home, voice assistant, online education, etc., and can generate voices of different character settings as needed, increasing the application scope and flexibility of voice generation.

[0060] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0062] Figure 1 A block diagram of a speech generating device according to an embodiment of the present disclosure is shown.

[0063] Figure 2 A block diagram of a speech generation module according to an embodiment of the present disclosure is shown.

[0064] Figure 3 A flowchart of a method for training a speech generating device according to an embodiment of the present disclosure is shown.

[0065] Figure 4 A flowchart of determining a spoken text corresponding to a sample audio according to an embodiment of the present disclosure is shown.

[0066] Figure 5 A diagram of a training architecture according to an embodiment of the present disclosure is shown.

[0067] Figure 6 A flow chart of a speech generation method according to an embodiment of the present disclosure is shown.

[0068] Figure 7 A block diagram of a speech generating device according to an embodiment of the present disclosure is shown.

[0069] Figure 8 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0070] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0071] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0072] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0073] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0074] In the disclosed embodiment, the spoken text rewriting module receives input prompt information and rewritten text, and rewrites the rewritten text into a spoken text corresponding to the prompt information according to the prompt information; the speech generation module generates a target speech corresponding to the spoken text based on the prompt information. Thus, the spoken text rewriting module can rewrite the rewritten text into a spoken text corresponding to the prompt information according to the prompt information, and further, the speech generation module can also generate a target speech corresponding to the spoken text based on the prompt information, so that the generated target audio can match the prompt information, that is, can match the user's needs, and the user controllability is high, which improves the audio generation effect.

[0075] Figure 1 A block diagram of a speech generating device according to an embodiment of the present disclosure is shown as follows: Figure 1As shown, the device 10 comprises:

[0076] The spoken text rewriting module 11 receives the inputted prompt information and the rewritten text, and rewrites the rewritten text into the spoken text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the spoken degree of the rewritten spoken text;

[0077] The speech generation module 12 generates a target speech corresponding to the spoken text based on the prompt information.

[0078] The prompt information is used to guide the text rewriting and speech generation process, and provides information about the expected colloquial level and style of the output text. The prompt information will include specific requirements for the colloquial rewriting text, such as emotional expression, speaking style, and scene adaptability. For example, the prompt information can be "a middle-aged male character in a dialogue scene, with colloquial expressions of relaxed and humorous emotions."

[0079] The rewritten text is the original text content that needs to be rewritten in a colloquial way. It can be a more formal or written text that has not been processed in a colloquial way. For example: "I think this issue should be viewed in this way."

[0080] The colloquial text is a text that has been rewritten to be more in line with spoken expression habits and contains more colloquial elements (such as modal particles, pauses, and emotional expressions). The rewritten text sounds more natural and closer to real conversations. For example: "Well, I think this question should be viewed this way."

[0081] The prompt information may include scene information and / or character information. The scene information is used to describe the text usage scenario, such as dialogue, audiobook, podcast, commentary, etc. It is used to guide the specific elements of the scene that should be considered when rewriting colloquial language. For example, "dialogue scene" means that the rewritten text should be more relaxed and natural, and contain more daily conversation elements.

[0082] Personality information is used to describe speaker characteristics, such as age, gender, personality, etc. When personality information is used to guide colloquial rewriting, the speaker's specific characteristics should be considered to produce text that matches his or her personality. For example, "middle-aged male personality" means that the rewritten text should reflect the speaking style and habits of middle-aged men.

[0083] The spoken text rewriting module can receive the input rewritten text, and then rewrite the rewritten text into spoken text according to the prompt information, that is, rewrite the text in combination with scene information and / or character information to make it more in line with spoken expression habits.

[0084] Then, the speech generation module generates the target speech corresponding to the colloquial text according to the prompt information. The speech generation module can receive the colloquial text output by the colloquial text rewriting module, and then generate speech output that meets the requirements according to the prompt information, combined with the scene information and / or character information, and the target timbre (such as reference audio).

[0085] In the disclosed embodiment, through the rewriting of colloquial text and the generation of voice based on prompt information, the generated voice is closer to real conversation, the emotional expression is richer, and the naturalness and anthropomorphism of the voice generation are improved. In addition, through the control of prompt information (including scene information and / or character information), the user can flexibly adjust the colloquial degree, emotional expression, etc. of the generated voice, enhance the controllability of voice generation, so as to be suitable for different voice interaction scenarios, such as smart home, voice assistant, online education, etc., and can generate voices of different character settings as needed, increasing the application scope and flexibility of voice generation.

[0086] In a possible implementation, the spoken text rewriting module includes a first large language model; the first large language model rewrites the rewrite text according to the scene information and / or character information in the prompt information to obtain the spoken text.

[0087] The first language model is the model responsible for actually rewriting the rewritten text according to the prompt information in the spoken text rewriting module. The first language model is a trained deep learning model that can understand and process natural language text, and generate text that meets the spoken requirements based on the scene information and / or character information in the input prompt information and the content of the rewritten text.

[0088] For example, a large language model based on the Transformer architecture can be fine-tuned (such as the LoRA method) to enable it to be rewritten in a colloquial manner for specific scene information and character information. For the specific training method for the first large language model, please refer to the possible implementation methods provided in this disclosure, which will not be described in detail here.

[0089] In the disclosed embodiment, the colloquial text rewriting module includes a first large language model; the rewritten text is rewritten by the first large language model according to the scene information and / or character information in the prompt information to obtain the colloquial text. Since the large language model has an excellent effect on text processing, the use of the large language model for colloquial rewriting can generate a more natural and vivid colloquial text, and by setting different prompt words, such as scene information and / or character information, the degree of colloquialism, tone, etc. can be controlled to adapt to the colloquial needs in different scenarios.

[0090] In a possible implementation, the spoken text contains emotional information and / or spoken events, and the emotional information and / or spoken events are represented based on emojis.

[0091] As an intuitive and easy-to-understand visual symbol, emoji is very suitable for representing complex emotions and spoken events. For example, a cough can be represented by the corresponding emoji of a cough, and a laugh can be represented by the corresponding emoji of a laugh. This representation method not only makes it easier and more intuitive to annotate data, but also makes full use of the emoji-related information that the large language model has learned during the pre-training stage.

[0092] Since emojis themselves are a widely understood visual language, annotators can easily understand and use them to annotate training data without additional training.

[0093] Furthermore, since emojis are already existing tokens in the large language model, they can be directly understood and generated by the model. Therefore, when fine-tuning the first large language model, there is no need to significantly modify or retrain the output layer of the model, saving time and resources.

[0094] In one possible implementation, the speech generation module includes: a voiceprint encoder, which encodes the reference audio corresponding to the personality information according to the personality information in the prompt information to obtain a voiceprint vector for characterizing the timbre of the personality information; and a second language model, which generates an audio code corresponding to the spoken text based on the voiceprint vector.

[0095] In a possible implementation, the speech generation module includes: an audio decoder that encodes the audio output by the second largest language model and decodes it into a target speech.

[0096] A voiceprint encoder is a coding tool that can convert audio information into a specific vector. By analyzing the timbre, pitch and other features in the audio, the voiceprint encoder extracts a feature vector that can uniquely or mainly represent the identity of a specific audio speaker, namely a voiceprint vector. In the speech generation module, the voiceprint encoder can be used to encode a speaker's reference audio (such as a sentence or a speech) into a voiceprint vector, which can then be used to guide the speech generation model to generate speech with the same or similar timbre as the speaker.

[0097] The second largest language model is a speech generation model based on the large language model technology. During the training process, the second largest language model learned the ability to convert spoken text into audio encoding. It receives voiceprint vectors and spoken text (including emotional information) as input and generates corresponding audio encoding.

[0098] The audio decoder is a tool that converts audio codes into playable audio. The audio decoder receives the audio code as input and restores the code to the original audio waveform or audio sampling points through a decoding algorithm, thereby generating playable speech. In the speech generation module, the audio decoder is used in conjunction with the second largest language model. After the speech generation model generates the audio code, the audio decoder decodes it into actual speech output.

[0099] In the speech generation task, the user may need to generate a speech with a voice similar to that of a specific speaker. Then, in this implementation, the voiceprint encoder can extract the voiceprint features in the audio and generate a voiceprint vector. The voiceprint encoder analyzes the reference audio and extracts the features that can characterize the voice of the speaker of the audio. These features are combined into a voiceprint vector. The voiceprint vector is the key to maintaining the consistency of the timbre in the subsequent speech generation process, and it will be used as one of the inputs of the speech generation model.

[0100] The second language model receives the spoken text (which may contain emotional information) and the voiceprint vector as input. Then, the model can generate the corresponding audio code based on the input text content and the set requirements of emotion, tone, etc., combined with the characteristics of the voiceprint vector.

[0101] When generating speech with specific emotional colors, the second largest language model can generate audio codes with corresponding emotional characteristics based on the input spoken text and emotional labels combined with the characteristics of the voiceprint vector.

[0102] After obtaining the audio code, the audio code can be restored to playable audio through the audio encoder. The audio decoder receives the audio code output by the second largest language model as input and restores the code to the original audio waveform or audio sampling point through the decoding algorithm. The decoded audio can be played directly or processed later.

[0103] In the disclosed embodiment, the voiceprint features of the reference audio are extracted through a voiceprint encoder to generate a voiceprint vector, which is used in the speech generation process, ensuring the consistency of the timbre of the generated speech and the reference audio, providing users with more choices and control, and making the generated speech more in line with the needs of actual application scenarios. The second largest language model is based on the large language model technology, and has learned the mapping relationship between spoken text and audio encoding through training. During the generation process, the model can generate audio encoding with corresponding characteristics based on the input spoken text and emotional tags and other information, thereby improving the naturalness and anthropomorphism of speech generation. By setting different emotional tags, spoken events and other information, the second largest language model can generate speech with different emotional colors.

[0104] like Figure 2 As shown, Figure 2A block diagram of a speech generation module according to an embodiment of the present disclosure is shown. In this embodiment, the speech generation module includes a voiceprint encoder, an autoregressive large language model (a second large language model), and an audio decoder.

[0105] In addition, the present disclosure also provides a method for training a speech generating device, which is used to train any speech generating device provided by the present disclosure.

[0106] Figure 3 A flow chart showing a method for training a speech generating device according to an embodiment of the present disclosure is shown. Figure 3 As shown, the method includes:

[0107] In step S21, the spoken text corresponding to the sample audio is determined;

[0108] In step S22, the sample audio is encoded to obtain audio code;

[0109] In step S23, a speech generation training set is constructed based on the spoken text and the audio code, and the second largest language model in the speech generation module is trained, wherein the spoken text serves as the input of the second largest language model, and the audio code serves as the expected output of the second largest language model.

[0110] In the disclosed embodiment, the second largest language model in the speech generation module is trained by the speech generation training set. The speech generation training set may include a spoken text T and an audio code C. The second largest language model in the speech generation module is trained by taking the spoken text T as the input of the second largest language model and taking the audio code as the expected output of the large language model.

[0111] For the spoken text in the speech generation training set, it can be obtained by determining the spoken text corresponding to the sample audio, converting the non-textual audio content into understandable text form, and ensuring that the text has spoken features. Specifically, speech recognition technology (such as ASR) can be used to convert the speech content in the audio into text. Then, through text rewriting technology or rules, colloquial elements (such as modal particles, pause markers, colloquial expressions, etc.) are inserted into the text to make the text more in line with daily spoken habits.

[0112] Suppose there is a sample audio content of "Haha, you are finally back." Through speech recognition technology, the audio is converted into text "Haha, you are finally back, cough cough". At the same time, when the acoustic event detects laughter and coughing, the text will be converted into " <laughter> <laughter>, you are finally back, <cough> <cough>".in, <laughter>and <cough>is the model input representation corresponding to laughter and cough. In this embodiment, laughter is used <laughter>,cough <cough>, scream <scream>, crying <cry>Four colloquial events, this rewriting makes the text more colloquial.

[0113] When the sample audio is encoded to obtain an audio code, the audio signal is converted into a data format that can be processed by a computer. This process involves the digitization and encoding of the audio signal. Digitization is the conversion of the audio signal from analog form to digital form, while encoding is the further processing of the digital signal into a format suitable for processing by a machine learning model. In this case, the audio may be encoded using a technique such as a vector quantization variational autoencoder (VQ-VAE) to obtain a series of discrete audio codes.

[0114] Suppose there is a sample audio. Through the VQ-VAE technology, the audio is encoded into a series of discrete audio coding units, such as "[audio_token1, audio_token2, ..., audio_tokenN]", where each audio_token represents a specific part or feature of the audio signal, and N is a positive integer.

[0115] A speech generation training set is constructed based on the spoken text and the audio encoding, and the second largest language model in the speech generation module is trained to learn the mapping relationship from text to audio. First, the spoken text and the corresponding audio encoding are used as training data pairs to construct a speech generation training set. Then, the training set is trained using the second largest language model so that the model can learn the mapping relationship from the spoken text to the audio encoding. During the training process, the model will continuously adjust its parameters to minimize the difference between the predicted audio encoding and the actual audio encoding.

[0116] Suppose there is a training set containing multiple spoken text-audio encoding pairs. For example, " <laughter> <laughter>, you are finally back, <cough> <cough>" corresponds to [audio_token1, audio_token2, ..., audio_tokenN]. After training the training set with the second largest language model, the model can learn to generate corresponding audio codes from similar spoken texts. When a new spoken text is input, the model can generate the corresponding audio code, which is then restored to an audio waveform through an audio decoder.

[0117] In traditional large language models, the input and output of the model are usually limited to text tokens. However, in the scenario of speech generation, the model is expected to directly output audio rather than just text. In order to achieve the conversion from text to audio, the audio information needs to be converted into a token form that the model can understand.

[0118] Since the large language model cannot process audio data directly, it is not possible to directly use audio as the expected output to supervise the training of the second largest speech model during training. Therefore, the sample audio is converted into tokens that the model can understand. These tokens constitute the audio encoding, which is used to verify the validity of the audio encoding output by the second largest language model during training.

[0119] In the disclosed embodiment, by converting the audio into an audio code and taking the spoken text as input, the large language model can learn the mapping relationship between the text and the audio code. After the training is completed, the model can generate the corresponding audio code according to the input spoken text, and then restore the audio code to an audio waveform through an audio decoder (such as Hifi-GAN), thereby realizing speech generation. This method not only improves the naturalness and anthropomorphism of speech generation, but also enables the model to process rich spoken information and generate more natural and fluent speech.

[0120] In one possible implementation, determining the spoken text corresponding to the sample audio includes: performing speech recognition on the sample audio to obtain recognized text; performing spoken event detection on the sample audio to obtain spoken events; performing emotion classification on the sample audio to obtain emotion information; and inserting the emotion information and the spoken events into the recognized text to obtain the spoken text.

[0121] A detailed description of this process can be found in Figure 4 , Figure 4 The flowchart of determining the spoken text corresponding to the sample audio according to the embodiment of the present disclosure is shown. First, the sample audio is subjected to automatic speech recognition (Automatic Speech Recognition, ASR). The speech content in the audio is automatically converted into text form, that is, the recognized text is obtained.

[0122] Then, the sample audio is subjected to spoken event detection, which can be achieved through acoustic event detection (AED). In order to find out various spoken events in the audio, such as laughter, coughing, screaming, crying, whispering, etc. These spoken events are non-verbal expressions that often appear in people's daily communication. They are essential for expressing emotions and the richness of context. The detected spoken events will be combined with the text previously recognized by ASR, and these events will be inserted into the appropriate positions in the text through preset rules, so as to obtain text containing spoken events.

[0123] At the same time, emotion recognition is used to classify the sample audio and determine the type of emotion expressed in the audio, such as happiness, sadness, anger, etc. Emotional information is another important dimension in speech generation, which can make the generated speech closer to the real expression of humans and enhance the emotional color and expressiveness of the speech.

[0124] Finally, the emotional information and spoken events obtained in the above steps are inserted into the initial recognized text to form the final spoken text. The spoken text not only contains the original speech content, but also incorporates spoken events and emotional information, making the text closer to the real oral communication scene.

[0125] After such a preprocessing process, each piece of training data can be expressed in the form of {T, A, e}, where A is audio, T is a text sequence containing spoken events, and e represents emotional information. This provides rich training materials for the subsequent speech generation model, helping the model learn more natural and realistic spoken expressions.

[0126] In a possible implementation, encoding the sample audio to obtain audio coding includes: encoding input audio to obtain an audio vector; and performing a vector quantization operation on the audio vector to obtain a discrete audio coding.

[0127] Before training the speech generation model, the audio is discretely encoded. This is because the input and output of traditional large language models are text tokens (i.e., the basic units of text, such as words, characters, etc.), and they cannot directly process or output audio data. In order to generate audio based on text, the audio data needs to be converted into a series of audio tokens so that the large language model can process it for subsequent training.

[0128] In this process, the audio needs to be discretized. Specifically, the audio data is originally composed of a series of continuous sampling points. For example, a 16 Hz audio has 16 audio sampling points in 1 second. These sampling points are continuous, but in order for the large language model to be able to process them, we need to convert them into discrete tokens.

[0129] This process can be implemented based on the Vector Quantization-Variational AutoEncoder (VQVAE) technology. VQVAE is a deep neural network model that can encode audio. The input of the network is an audio sequence. After passing through the encoder, the audio data is converted into a series of vectors (i.e., audio vectors). Then, these audio vectors are vectorized to obtain discrete audio codes (i.e., audio tokens).

[0130] Vector Quantization (VQ) is a technique for converting continuous data into discrete data. In VQVAE, it is implemented by mapping the audio vector to the nearest neighbor vector in a predefined codebook. This codebook contains a series of discrete vectors, each of which represents a possible audio feature. By selecting the vector in the codebook that is closest to the audio vector, we can convert the continuous audio vector into a discrete audio code.

[0131] Figure 5 The training architecture diagram according to the embodiment of the present disclosure is shown. When training the audio encoder and vector quantization parameters, the following can be constructed: Figure 5 In the training architecture shown, after the audio data passes through the audio encoder and completes vector quantization, the quantized result needs to be decoded to obtain the output audio. This process is achieved through the VQVAE decoder. The input of the decoder is the quantized audio code, and after decoding, the output is audio data that is as similar as possible to the input audio.

[0132] During training, we continuously adjust the parameters of the VQVAE network to minimize the error between the output audio and the input audio. In this way, after training, we can use this network to encode any audio, that is, input any audio, use the result after the intermediate layer VQ as the discrete encoding of the audio, and use it as the true value of the audio encoding output by the second largest language model.

[0133] Similarly, in the inference phase, after the second largest language model outputs discrete audio codes, the codes can be restored to audio waveforms through other vocoders (such as Hifi-GAN, etc.). In this way, we have realized the conversion process from text tokens to audio tokens.

[0134] In the embodiment of the present disclosure, continuous audio data is converted into discrete audio codes so that a large language model can process it.

[0135] In a possible implementation, the speech generation training set also includes a reference audio s, the timbre of the reference audio is the same as the timbre of the sample audio, and the reference audio is used to control the timbre of the audio generated by the speech generation module.

[0136] Here, the reference audio s may be part of the audio generation training set, and the reference audio may come from another audio belonging to the same speaker as the target audio A (i.e., the original audio or sample audio corresponding to the audio to be generated). The main function of the reference audio is to control the timbre of the audio generated by the speech generation module.

[0137] In one possible implementation, the speech generation training set can be represented as {T, A, e, C, s}, where:

[0138] T: Indicates spoken text, including spoken events.

[0139] A: Target audio, that is, the original audio corresponding to the audio whose timbre you want to generate.

[0140] e: Emotion classification, indicating the emotional information corresponding to audio A.

[0141] C: The encoded audio sequence, that is, the result after audio A is processed by the audio encoder, which is the expected output of the second largest language model.

[0142] s: reference audio, used to control the timbre of the output audio, comes from another audio from the same speaker as A.

[0143] Training of speech generation model:

[0144] When training a speech generation model, its input includes reference audio s, sentiment classification e, and representation of spoken text T. Reference audio s: As mentioned above, it is used to control the timbre of the output audio. Sentiment classification e: It represents the emotion corresponding to the target audio A, which guides the model on which emotion should be expressed when generating audio.

[0145] Representation of spoken text T: Replace text and spoken events with specific representation units and convert them into tokens that the model can understand. These tokens include text tokens such as Chinese characters, English BPE, and laughter. <laughter>,cough <cough>, scream <scream>Colloquial event tokens such as

[0146] The output of the model is the predicted audio encoding C after encoding. Here, C is the encoded audio sequence predicted by the model, and it should be as close as possible to the true audio encoding C. During training, the goal is to make the audio encoding predicted by the model as close as possible to the true audio encoding. In this process, the model needs to learn how to generate an output similar to the true audio encoding C from the representations of the given reference audio s, emotion classification e, and colloquial text T.

[0147] In the embodiments of the present disclosure, through the reference audio s, the model can maintain the same timbre as the target audio A when generating audio, thereby improving the naturalness and anthropomorphism of speech generation. By combining the representations of the emotion classification e and the colloquial text T, the model can generate audio with rich emotions and colloquial features, which is of great significance for improving the speech generation effect in scenarios such as human-computer interaction.

[0148] In a possible implementation, the second large language model can be an autoregressive large language model. When training the autoregressive large language model, when the large language model is an autoregressive large language model, its training process aims to make the "predicted text-audio" pairs output by the model as similar as possible to the "true text-audio" pairs.

[0149] Specifically, the input text is split into individual characters or sub-word units (which may be Chinese characters for Chinese and units after BPE tokenization for English) and input one by one. For example, for the input text "The weather is really nice today!", the model does not receive the entire sentence at once, but receives it character by character in order.

[0150] The model works in an autoregressive manner, that is, it predicts only the next character or sub-word unit each time. When the input is "今 (jīn)", the model predicts and outputs "天 (tiān)". Then, taking "天 (tiān)" as the new input, the model predicts and outputs the next character "气 (qì)". This cycle continues until the last character "! (!)" of the entire sentence "The weather is really nice today!" is input. After inputting "! (!)", the model outputs a special end symbol " <eos>”, indicating the end of the text prediction stage.

[0151] After the text prediction phase, the model enters the audio prediction phase. At this point, the model input becomes the end symbol output by the previous text prediction phase. <eos>". The model starts predicting and outputting the first audio token based on the previously predicted text and possible contextual information (such as sentiment tags, voiceprint features, etc.). Then, the first audio token is used as a new input, and the model predicts and outputs the next audio token. This cycle continues until the model outputs another special end symbol" <eoa>”, indicating the end of the audio prediction phase.

[0152] Throughout the training process, the goal of the model is to make the predicted "predicted text-audio" sequence as similar as possible to the real "real text-audio" sequence. This is usually achieved by calculating a certain distance or loss function between the predicted sequence and the real sequence, such as cross entropy loss. By adjusting the parameters of the model and minimizing this loss function, an autoregressive large language model that can accurately predict text and audio is trained.

[0153] In one possible implementation, a spoken text rewriting module is trained based on the spoken rewritten text, the text before rewriting and the prompt words, wherein the text before rewriting and the prompt words serve as inputs of the spoken text rewriting module, and the spoken rewritten text serves as the expected output of the spoken text rewriting module.

[0154] In this implementation, in order to train the colloquial text rewriting module, a strategy based on colloquial rewriting text, the text before rewriting (original text) and prompt words is adopted. The input of the colloquial rewriting text model is: the text before rewriting (T i ) and the prompt word (p). The expected output is: the colloquial rewrite text (T o ). That is, each training corpus includes {p,T i ,T o }.

[0155] The spoken text rewriting module can be a large language model. In the disclosed embodiment, an open source large language model is used as the basic model, and the LoRA method is used to fine-tune the model so that the model parameters are given prompt words and input text, and the output result is as close as possible to the target rewritten text. The large language model has an excellent effect on text processing and can generate more natural and vivid spoken text. By using a large language model for spoken rewriting, its powerful text generation and context understanding capabilities can be fully utilized to generate spoken text that conforms to specific scenarios and personalities.

[0156] When collecting training data, the training data needs to be manually annotated to ensure accuracy and diversity. When annotating, different scenarios (such as dialogues, audiobooks, podcasts, narration, etc.) and corresponding personas (such as the speaker's age and gender) are considered. There are about 100 training data for each combination of persona and scenario.

[0157] Let's take a specific example to illustrate the training process:

[0158] Prompt: "Assume you are a 20-year-old woman talking to someone in a podcast setting. Please rewrite the input text to make it more consistent with your speaking style."

[0159] Input text: "I think this problem should be viewed in this way."

[0160] Rewritten text: "Well, I think this is the way to look at this issue."

[0161] Through such a training set, the colloquial text rewriting module can learn how to rewrite the original text in a colloquial way under different scenarios and character settings.

[0162] During the training process, the spoken text rewriting module will try to generate an output that is as similar as possible to the expected spoken rewriting text based on the input pre-rewriting text and prompt words. Through continuous iteration and optimization, the spoken text rewriting module can gradually improve the accuracy and naturalness of spoken rewriting.

[0163] In order to enable the large language model to output the rewritten text, this embodiment uses emoji to represent emotional tags and colloquial events. For example, when the input text is "Haha, you are finally back.", the rewritten text may be "smiley emoji, you are finally back, cough emoji." The rewritten text needs to be converted into a representation unit sequence and fed into the speech generation model. For example, the representation unit sequence corresponding to the text in the above example is " <laughter> <laughter>, you are finally back, <cough> <cough>. ".

[0164] In one possible implementation, the speech generation method can be executed by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be executed by a processor calling computer-readable instructions stored in a memory.

[0165] In addition, the present disclosure also provides a speech generation method, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any speech generation device method provided by the present disclosure. The corresponding technical solutions and descriptions refer to the corresponding records in the device part and will not be repeated here.

[0166] Figure 6 A flow chart of a speech generation method according to an embodiment of the present disclosure is shown as follows: Figure 6 As shown, the method includes:

[0167] In step S31, receiving inputted prompt information and rewritten text, and rewriting the rewritten text into a colloquial text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the colloquial degree of the rewritten colloquial text;

[0168] In step S32, Based on the prompt information, generate the target speech corresponding to the colloquial text .

[0169] In a possible implementation, the spoken text rewriting module includes a first large language model;

[0170] The first language model rewrites the rewritten text according to the scene information and / or character information in the prompt information to obtain a spoken text.

[0171] In a possible implementation, the spoken text contains emotional information and / or spoken events, and the emotional information and / or spoken events are represented based on emojis.

[0172] In a possible implementation, generating a target speech corresponding to the colloquial text based on the prompt information includes:

[0173] According to the character information in the prompt information, encoding the reference audio corresponding to the character information to obtain a voiceprint vector for representing the timbre of the character information;

[0174] The second largest language model is used to generate an audio code corresponding to the spoken text based on the voiceprint vector.

[0175] In a possible implementation, generating a target speech corresponding to the colloquial text based on the prompt information includes:

[0176] Encode the audio output of the second largest language model and decode it into the target speech.

[0177] According to one aspect of the present disclosure, a training device for a speech generating device is provided, for training the above-mentioned speech generating device.

[0178] Figure 7 A block diagram of a speech generating device according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the device 40 includes:

[0179] A spoken text determination module 41, used to determine the spoken text corresponding to the sample audio;

[0180] The encoding module 42 is used to encode the sample audio to obtain audio code;

[0181] The first training module 43 is used to construct a speech generation training set based on the spoken text and the audio code, and train the second largest language model in the speech generation module, wherein the spoken text serves as the input of the second largest language model, and the audio code serves as the expected output of the second largest language model.

[0182] In a possible implementation, the spoken text determination module is used to:

[0183] Performing speech recognition on the sample audio to obtain recognized text;

[0184] Performing spoken event detection on the sample audio to obtain spoken events;

[0185] Performing emotion classification on the sample audio to obtain emotion information;

[0186] The emotional information and the spoken events are inserted into the recognized text to obtain the spoken text.

[0187] In a possible implementation, the encoding module is used to:

[0188] Encode the input audio to obtain an audio vector;

[0189] A vector quantization operation is performed on the audio vector to obtain discrete audio code.

[0190] In a possible implementation, the speech generation training set also includes reference audio, the timbre of the reference audio is the same as the timbre of the sample audio, and the reference audio is used to control the timbre of the audio generated by the speech generation module.

[0191] In a possible implementation, the training device further includes:

[0192] The second training module is used to train the spoken text rewriting module based on the spoken rewritten text, the text before rewriting and the prompt words, wherein the text before rewriting and the prompt words serve as inputs of the spoken text rewriting module, and the spoken rewritten text serves as the expected output of the spoken text rewriting module.

[0193] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0194] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0195] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0196] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0197] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0198] Figure 8 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0199] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0200] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0201] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0202] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0203] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0204] The computer program instructions for performing the disclosed operation may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0205] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0206] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0207] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0208] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0209] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0210] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0211] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0212] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0213] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.< / cough> < / cough> < / laughter> < / laughter> < / eoa> < / eos> < / eos> < / scream> < / cough> < / laughter> < / cough> < / cough> < / laughter> < / laughter> < / cry> < / scream> < / cough> < / laughter> < / cough> < / laughter> < / cough> < / cough> < / laughter> < / laughter>

Claims

1. A speech generating device, characterized in that: include: A spoken text rewriting module receives inputted prompt information and a rewritten text, and rewrites the rewritten text into a spoken text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the spoken degree of the rewritten spoken text; The speech generation module generates a target speech corresponding to the spoken text based on the prompt information.

2. The device according to claim 1, characterized in that The spoken text rewriting module includes a first language model; The first language model rewrites the rewritten text according to the scene information and / or character information in the prompt information to obtain a spoken text.

3. The device according to claim 1, characterized in that The spoken text contains emotional information and / or spoken events, and the emotional information and / or spoken events are expressed based on emoticons emoji.

4. The device according to claim 1, characterized in that The speech generation module comprises: A voiceprint encoder, which encodes the reference audio corresponding to the personality information according to the personality information in the prompt information to obtain a voiceprint vector for representing the timbre of the personality information; The second language model generates an audio code corresponding to the spoken text based on the voiceprint vector.

5. The device according to claim 4, characterized in that The speech generation module comprises: The audio decoder encodes the audio output by the second largest language model and decodes it into the target speech.

6. A method for training a speech generating device, characterized in that: For training the apparatus as claimed in any one of claims 1 to 5, the method comprising: Determine the spoken text corresponding to the sample audio; Encoding the sample audio to obtain audio code; A speech generation training set is constructed based on the spoken text and the audio code, and a second largest language model in the speech generation module is trained, wherein the spoken text serves as an input of the second largest language model, and the audio code serves as an expected output of the second largest language model.

7. The method according to claim 6, characterized in that The step of determining the spoken text corresponding to the sample audio includes: Performing speech recognition on the sample audio to obtain recognized text; Performing spoken event detection on the sample audio to obtain spoken events; Performing emotion classification on the sample audio to obtain emotion information; The emotional information and the spoken events are inserted into the recognized text to obtain the spoken text.

8. The method according to claim 6, characterized in that The step of encoding the sample audio to obtain audio encoding comprises: Encode the input audio to obtain an audio vector; A vector quantization operation is performed on the audio vector to obtain discrete audio code.

9. The method according to claim 6, characterized in that The speech generation training set also includes reference audio, the timbre of the reference audio is the same as the timbre of the sample audio, and the reference audio is used to control the timbre of the audio generated by the speech generation module.

10. The method according to claim 6, characterized in that The method further comprises: Based on the colloquial rewritten text, the text before rewriting and the prompt words, the colloquial text rewriting module is trained, wherein the text before rewriting and the prompt words serve as inputs of the colloquial text rewriting module, and the colloquial rewritten text serves as the expected output of the colloquial text rewriting module.

11. A speech generation method, characterized in that: include: receiving inputted prompt information and rewritten text, and rewriting the rewritten text into a colloquial text corresponding to the prompt information according to the prompt information, wherein the prompt information is used to control the colloquial degree of the rewritten colloquial text; Based on the prompt information, a target speech corresponding to the colloquial text is generated.

12. A training device for a speech generating device, characterized in that: Used for training the speech generation device according to any one of claims 1 to 6, the training device comprising: A spoken text determination module, used to determine the spoken text corresponding to the sample audio; An encoding module, used for encoding the sample audio to obtain audio code; The first training module is used to construct a speech generation training set based on the spoken text and the audio code, and train the second largest language model in the speech generation module, wherein the spoken text serves as the input of the second largest language model, and the audio code serves as the expected output of the second largest language model.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to implement the method described in any one of claims 6 to 11.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 6 to 11 is implemented.

Images