Speech generation method, device, computer equipment and storage medium

By uniformly quantizing and splicing multimodal audio data and inputting a speech generation large model, the problem of low robustness and sound quality defects caused by independent modeling of speech generation models in the existing technology is solved, and multiple speech generation tasks are realized. Integration and performance improvement.

CN118398001BActive Publication Date: 2024-11-08DMAI (GUANGZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410823160.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-25
Publication Date
2024-11-08
Estimated Expiration
2044-06-25

AI Technical Summary

Technical Problem

In the prior art, due to independent modeling and small data scale, the speech generation model leads to incomplete style retention during speech conversion, and the sound quality of the synthetic audio has many flaws and is less robust.

Method used

By obtaining multi-modal audio data (such as audio, text, and midi data), performing unified quantization, generating quantization sequences, and splicing them according to the target speech generation task, inputting pre-trained speech generation large model, generating predicted speech sequences, and realizing multiple Integration of speech generation tasks.

Benefits of technology

It effectively improves the performance limit and robustness of speech generation tasks, realizes the integration of multiple tasks, and does not require independent modeling of multiple speech generation tasks, and significantly improves the audio quality and style retention of generated audio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118398001B_ABST
    Figure CN118398001B_ABST
Patent Text Reader

Abstract

The present invention discloses a speech generation method, device, computer equipment and storage medium, and relates to the field of large models. The method comprises: obtaining multimodal audio data, the multimodal audio data comprises one or more of the following: audio data, text data, midi data, phoneme data; uniformly quantizing the multimodal audio data to obtain at least one quantization sequence corresponding to different modes; splicing at least one quantization sequence according to a target speech generation task to obtain a target quantization sequence, the target speech generation task comprises one or more of the following: TTA task, SVS task, VC task, SE task, TTS task; inputting the target quantization sequence and the conditional sequence into a pre-trained speech generation large model to generate a predicted speech sequence. There is no need to independently model multiple speech generation tasks, and the integration of multiple speech tasks can be achieved, which can effectively improve the performance upper limit and robustness of various speech generation tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of large models, and in particular to a speech generation method, apparatus, computer equipment and storage medium. Background Art

[0002] Speech generation is an important part of artificial intelligence (AI), including many tasks such as text to speech (TTS), voice conversion (VC), singing voice synthesis (SVS), and text to audio (TTA). Currently, small models are generally used to independently build models for each task. However, the upper limit of the model is not high and the robustness is average when training models with small-scale data. For example, the style retention is incomplete during speech conversion and there are many defects in the sound quality of synthesized audio when training with small data volume. Summary of the invention

[0003] The object of the present invention is to provide a speech generation method, apparatus, computer equipment and storage medium.

[0004] In a first aspect, the present invention provides a speech generation method, the method comprising:

[0005] Acquire multimodal audio data, wherein the multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data;

[0006] Performing unified quantization on the multimodal audio data to obtain at least one quantization sequence corresponding to different modes;

[0007] According to a target speech generation task, the at least one quantization sequence is spliced ​​to obtain a target quantization sequence, wherein the target speech generation task includes one or more of the following: a non-human voice synthesis TTA task, a music generation SVS task, a speech conversion VC task, a speech noise reduction SE task, and a speech synthesis TTS task;

[0008] The target quantization sequence and the conditional sequence are input into a pre-trained speech generation model to generate a predicted speech sequence.

[0009] Optionally, the uniformly quantizing the multimodal audio data to obtain at least one quantization sequence corresponding to different modalities includes:

[0010] For unimodal data of any mode in the multimodal audio data, input the unimodal data into a first encoder to obtain continuous latent features of each frame corresponding to the unimodal data;

[0011] Performing residual vector quantization RVQ on the latent features corresponds to a first quantization sequence of the unimodal data, wherein the first quantization sequence is to obtain N-bit discrete audio features, where N is greater than zero.

[0012] Optionally, the method comprises:

[0013] Inputting the first quantized sequence into a first decoder for reconstruction to obtain a first reconstructed audio corresponding to the single modal data;

[0014] Inputting the first reconstructed audio and the unimodal data into a discriminator to obtain a discrimination result;

[0015] According to the determination result, parameters of the first decoder and / or the first encoder are updated.

[0016] Optionally, the conditional sequence is determined according to the text data, the speech generation model includes at least a second encoder and a second decoder, and the target quantization sequence and the conditional sequence are input into the pre-trained speech generation model to generate a predicted speech sequence, including:

[0017] Inputting the text data into a semantic recognition model to obtain the conditional sequence corresponding to the text data;

[0018] Inputting the condition sequence and the target quantization sequence into the second encoder for processing, thereby obtaining a first feature corresponding to the condition sequence and a second feature corresponding to the target quantization sequence;

[0019] Adding the first feature and the second feature in a unified dimension to obtain a fusion feature;

[0020] The fused features are input into the second decoder to generate a predicted speech sequence.

[0021] Optionally, the second encoder includes a global encoding unit and a local encoding unit, and the step of inputting the fused features into a second decoder to generate a predicted speech sequence includes:

[0022] Inputting the fused features into the global encoding unit to obtain feature distribution information;

[0023] The feature distribution information and the fusion feature are input into the local encoding unit to obtain the predicted speech sequence.

[0024] Optionally, the second encoder includes a linear layer, and the step of adding the first feature and the second feature in a uniform dimension to obtain a fused feature includes:

[0025] The first feature and the second feature are input into the linear layer to obtain the fused feature.

[0026] Optionally, each element in each of the quantized sequences corresponds to a vocabulary identifier in a prediction vocabulary, and each speech generation task corresponds to a vocabulary identifier in the prediction vocabulary;

[0027] The step of splicing the at least one quantization sequence according to the target speech generation task to obtain a target quantization sequence includes:

[0028] The lexical identifier corresponding to the target speech generation task and the lexical identifier corresponding to each element in each of the quantization sequences are concatenated to obtain the target quantization sequence;

[0029] The target quantization sequence further includes a start tag and an end tag corresponding to the target speech generation task, and a start tag and an end tag corresponding to each of the quantization sequences.

[0030] In a second aspect, the present invention provides a speech generating device, the device comprising:

[0031] An acquisition module, used for acquiring multimodal audio data, wherein the multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data;

[0032] A quantization module, used to uniformly quantize the multimodal audio data to obtain at least one quantization sequence corresponding to different modes;

[0033] A splicing module is used to splice the at least one quantization sequence according to a target speech generation task to obtain a target quantization sequence, wherein the target speech generation task includes one or more of the following: a non-human voice synthesis TTA task, a music generation SVS task, a speech conversion VC task, a speech noise reduction SE task, and a speech synthesis TTS task;

[0034] The prediction module is used to input the target quantization sequence and the conditional sequence into a pre-trained speech generation model to generate a predicted speech sequence.

[0035] In a third aspect, the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the speech generation method described in the first aspect.

[0036] In a fourth aspect, the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the speech generation method described in the first aspect.

[0037] Compared with the prior art, the beneficial effects provided by the present invention include: the target quantization sequence can be obtained by uniformly quantizing multimodal audio data, and the quantization sequences corresponding to the data of different modalities are spliced ​​with the target speech generation task, and then the target quantization sequence is input into the speech generation model to predict the generation results of various speech generation tasks. There is no need to independently model the various speech generation tasks, and the integration of various speech tasks can be achieved, which can effectively improve the performance ceiling and robustness of various speech generation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0039] Figure 1 A flowchart of a speech generation method provided by an embodiment of the present disclosure;

[0040] Figure 2 A block diagram of a speech generating device provided in an embodiment of the present disclosure;

[0041] Figure 3 A structural block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0044] In order to solve the technical problems in the aforementioned background technology, Figure 1 The following is a flow chart of a speech generation method provided by an embodiment of the present disclosure. The speech generation method is described in detail below. The method can be applied to electronic devices, such as computer devices or servers, which have information processing capabilities, such as Figure 1 As shown, the method includes:

[0045] Step S101, acquiring multimodal audio data.

[0046] The multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data.

[0047] It is understandable that audio data of different modalities can be used for different speech generation tasks, or for the same speech generation task. Among them, phonemes are the smallest units of human pronunciation, and tasks such as speech synthesis and singing synthesis use phoneme data as input. For tasks such as non-human voice synthesis and music generation, text data can be used as auxiliary information. Midi data can be a necessary input for singing synthesis, containing notes and their corresponding duration information. For generation tasks such as speech conversion and speech denoising, audio data can be used as the main input. In addition, in tasks such as style transfer or speech synthesis / speech conversion, reference audio can be used as additional input to provide information such as style or timbre.

[0048] Step S102: uniformly quantize the multimodal audio data to obtain at least one quantization sequence corresponding to different modes.

[0049] In some embodiments, audio quantization may be performed using a VQ-GAN approach, which will be described in detail in the following embodiments and will not be elaborated upon here.

[0050] Step S103, according to the target speech generation task, the at least one quantization sequence is spliced ​​to obtain a target quantization sequence, and the target speech generation task includes one or more of the following: non-human synthesis TTA task, music generation SVS task, speech conversion VC task, speech noise reduction SE task, and speech synthesis TTS task.

[0051] In some embodiments, the task names may be uniformly added to the vocabulary, which may include two labels: task start and task end. Then, the quantization sequence of each modality in the task may be concatenated into a complete sequence as the conditional input of the model.

[0052] Step S104, inputting the target quantization sequence and the conditional sequence into a pre-trained speech generation model to generate a predicted speech sequence.

[0053] In some embodiments, the target quantization sequence and the conditional sequence may be input into a pre-trained speech generation model to generate a corresponding predicted speech sequence, thereby achieving speech generation.

[0054] Among them, the large speech generation model can be trained and optimized using the cross entropy loss between the predicted speech sequence and the target quantized sequence. Similarly, in order for the model to have super generalization and robustness, hundreds of thousands of hours of speech data from various speech generation tasks can be used for model pre-training. For tasks such as zero-shot TTS, the pre-trained model can be used directly without any fine-tuning; for tasks such as standard TTS, if the timbre and style need to be further improved, the above pre-training process can be further used to update the model parameters using fine-tuning training data.

[0055] In some embodiments, the speech generation model is trained based on a sample quantization sequence and a sample condition sequence, and a sample speech sequence corresponding to the sample quantization sequence and the sample condition sequence. The formats of the sample quantization sequence and the sample condition sequence are respectively the same as the formats of the target quantization sequence and the condition sequence. The speech generation model can generate a predicted speech sequence based on the sample quantization sequence and the sample condition sequence, and calculate a loss function based on the predicted speech sequence and the sample speech sequence to iterate the parameters of the speech generation model until a trained speech generation model is obtained.

[0056] In the disclosed embodiments, the multimodal audio data can be uniformly quantized, and the quantization sequences corresponding to the data of different modalities can be concatenated with the target speech generation task to obtain the target quantization sequence, which can then be input into a large speech generation model to predict the generation results of multiple speech generation tasks. This eliminates the need to independently model the multiple speech generation tasks, and enables the integration of multiple speech tasks, which can effectively improve the performance ceiling and robustness of various speech generation tasks.

[0057] In some optional embodiments, the uniformly quantizing the multimodal audio data to obtain at least one quantization sequence corresponding to different modalities includes:

[0058] For unimodal data of any mode in the multimodal audio data, input the unimodal data into a first encoder to obtain continuous latent features of each frame corresponding to the unimodal data;

[0059] Residual Vector Quantization (RVQ) is performed on the latent feature, corresponding to a first quantization sequence of the single-modal data, wherein the first quantization sequence is to obtain N-bit discrete audio features, where N is greater than zero.

[0060] In some optional embodiments, the method includes:

[0061] Inputting the first quantized sequence into a first decoder for reconstruction to obtain a first reconstructed audio corresponding to the single modal data;

[0062] Inputting the first reconstructed audio and the unimodal data into a discriminator to obtain a discrimination result;

[0063] According to the determination result, parameters of the first decoder and / or the first encoder are updated.

[0064] In some optional embodiments, the conditional sequence is determined according to the text data, the speech generation model includes at least a second encoder and a second decoder, and the target quantization sequence and the conditional sequence are input into the pre-trained speech generation model to generate a predicted speech sequence, including:

[0065] Inputting the text data into a semantic recognition model to obtain the conditional sequence corresponding to the text data;

[0066] Inputting the condition sequence and the target quantization sequence into the second encoder for processing, thereby obtaining a first feature corresponding to the condition sequence and a second feature corresponding to the target quantization sequence;

[0067] Adding the first feature and the second feature in a unified dimension to obtain a fusion feature;

[0068] The fused features are input into the second decoder to generate a predicted speech sequence.

[0069] In some optional embodiments, the second encoder includes a global encoding unit and a local encoding unit, and the step of inputting the fused features into a second decoder to generate a predicted speech sequence includes:

[0070] Inputting the fused features into the global encoding unit to obtain feature distribution information;

[0071] The feature distribution information and the fusion feature are input into the local encoding unit to obtain the predicted speech sequence.

[0072] In some optional embodiments, the second encoder includes a linear layer, and the step of adding the first feature and the second feature in a uniform dimension to obtain a fused feature includes:

[0073] The first feature and the second feature are input into the linear layer to obtain the fused feature.

[0074] In some optional embodiments, each element in each of the quantized sequences corresponds to a vocabulary identifier in a prediction vocabulary, and each speech generation task corresponds to a vocabulary identifier in the prediction vocabulary;

[0075] The step of splicing the at least one quantization sequence according to the target speech generation task to obtain a target quantization sequence includes:

[0076] The lexical identifier corresponding to the target speech generation task and the lexical identifier corresponding to each element in each of the quantization sequences are concatenated to obtain the target quantization sequence;

[0077] The target quantization sequence further includes a start tag and an end tag corresponding to the target speech generation task, and a start tag and an end tag corresponding to each of the quantization sequences.

[0078] In order to enable those skilled in the art to better understand the overall technical solution provided by the embodiments of the present disclosure, the present disclosure also provides the following embodiments of a speech generation system, which may include a quantization module, a task module, an input module, and a prediction module.

[0079] For the quantization module, it can uniformly quantize the input of any speech generation task. For example, the input representations of various tasks are summarized into a unified vocabulary, and each token in any input sequence can be represented by a token in the vocabulary. Therefore, the core of the quantization module is to discretize the input of each module. The following introduces some discretization representation methods for task inputs. For other niche tasks, quantization can be performed according to the mainstream quantization methods introduced:

[0080] Phonemes are the smallest unit of human pronunciation. They are used as input for tasks such as speech synthesis and singing synthesis. Generally, phonemes need to be obtained from text. For texts in Latin languages ​​such as English and French, converting them to phonemes is relatively simple and can be done directly according to the vocabulary table. For Chinese texts, the process is more complicated and requires multiple steps such as word segmentation, polyphone disambiguation, rhythm prediction, and Chinese character to pinyin conversion.

[0081] Text, for tasks such as TTA (non-human voice synthesis) and SVS (music generation) that use text as prompt words, in theory, the vocabulary corresponding to the text can be aggregated to form a unified vocabulary. Considering the problem of model parameter quantity and the current research progress of language models, the present invention does not unify the text vocabulary, but uses the embedding generated by the pre-trained language model to represent the text, and the continuous text representation is used as auxiliary information as the model input.

[0082] Midi is a necessary input for singing synthesis, which contains notes and their corresponding duration information. The notes themselves are discrete tokens, which we can directly add to the unified vocabulary. To simplify the representation of note and duration sequences, we expand the notes by integer multiples according to the duration.

[0083] Audio, for VC (voice conversion), SE (voice noise reduction) and other generation tasks, audio is required as the main input. In addition, in style transfer or zero-shot TTS / VC and other tasks, reference audio is used as an additional input to provide style or timbre information. Converting continuously represented audio into discrete audio features is the key to the entire speech generation system. The audio target sequence is the sequence after audio quantization. Audio quantization determines the upper limit of the audio generated by the entire generation system. If the audio quantization model is not good, then the audio after the target sequence is dequantized will inevitably have many flaws.

[0084] Optionally, the VQ-GAN method can be used for audio quantization. The input audio passes through the encoder to obtain continuous hidden features of several speech frames. RVQ (Residual Vector Quantization) converts the continuous hidden features of each frame into N-bit discrete audio features. The decoder attempts to reconstruct the input audio using discrete features. The discriminator assists the quantization model in reconstructing the audio with high quality by judging whether the audio is real or generated. In this embodiment, the number of tokens per frame N can be 4. Although the higher N is, the better the reconstruction effect is, the audio sequence also increases linearly, and the prediction overhead of the subsequent model also increases accordingly.

[0085] In order to improve the upper limit and robustness of the speech generation system, this embodiment can use ultra-large-scale data to train the audio quantization model. After integrating the currently open-source TTS data, speech recognition data, music data and other types of data with hundreds of thousands of hours of data for training, good audio quantization and dequantization effects are achieved for various audio test data. After the model is trained, the audio is input to the encoder and discretized by RVQ to obtain the audio quantization features. The decoder and discriminator do not need to be used when performing audio quantization.

[0086] For the task module and the input module, the task determines the input of the model and its predicted target. Therefore, we first need to define the task type to facilitate the model to distinguish. Specifically, this embodiment adds the task name to the vocabulary (including two labels: task start and task end), and then splices the quantized sequence of each modality in the task into a complete sequence as the conditional input of the model. Before splicing, we also define the modal start label and modal end label for the input of each modality to distinguish the data type of each modality, and add the start label and end label of each modality to the global vocabulary. Specifically, taking the SVS (singing voice synthesis) task as an example, the input includes lyrics (lyrics can be further converted into phonemes), midi, reference audio (providing timbre style information) and target audio, then the format of the entire sequence after splicing can be:

[0087] “<SVS_task_beg>

[0088] <phoneme_beg> phoneme seq<phoneme_end>

[0089] <midi_beg> midi seq<midi_end>

[0090] <audio_beg> reference audio seq<audio_end>

[0091] <audio_beg> target audio seq<audio_end>

[0092] <SVS_task_end> ”.

[0093] For the prediction module, it aims to use the large model to combine the conditional sequence and other additional conditions as well as the target sequence to predict the target sequence in the form of next token prediction. The overall framework of the speech generation large model designed in this embodiment can include two parts: Encoder and Decoder. Encoder is responsible for fusing all conditional information and providing it to Decoder. Decoder predicts the target sequence based on the conditional information and the target sequence by autoregression.

[0094] The encoder consists of multiple layers of transformers, which include two types of input: one is the discrete sequence representation, such as MIDI, audio quantization sequence, etc. The encoder converts the discrete sequence into the corresponding embedding; the other is other optional continuous feature representations, such as the music generation task for a given prompt word, which uses a pre-trained language model to obtain the continuous feature representation of the text. The two features are dimensionally unified and added, and then sent to the subsequent transformer layer of the encoder to obtain the fused input features. Dimensional unification can be achieved by a linear layer.

[0095] The decoder takes into account that the audio quantization features have clear global and local characteristics, that is, there is a strong global dependency between speech frames, and each label in a speech frame has an obvious local dependency. For this reason, this embodiment designs the decoder as a global decoder plus a local decoder structure. The global decoder first roughly predicts the distribution of the entire target sequence based on the target sequence and the output of the encoder in a next-token prediction manner for reference by the local decoder. The local decoder accepts the output of the global decoder, the output of the encoder, and the token of each frame in the target sequence as input, and also predicts all the labels in the frame in a next-token prediction manner. The global decoder and the local decoder are also composed of multiple transformer blocks.

[0096] The entire speech generation model is trained and optimized using the cross entropy loss between the prediction sequence and the target sequence. Similarly, in order for the model to have super generalization and robustness, this embodiment uses hundreds of thousands of hours of speech data from various speech generation tasks for model pre-training. For tasks such as zero-shot TTS, the pre-trained model can be used directly without any fine-tuning; for tasks such as standard TTS, if the timbre and style need to be further improved, the above pre-training process can be further used to update the model parameters using fine-tuning training data.

[0097] In the disclosed embodiments, the integration of various speech generation tasks such as speech synthesis and speech conversion is achieved through unified training, and the prior knowledge between various tasks is fully utilized to generate the desired target audio with high quality, which effectively solves the current problem of independent modeling for each generation task, and can significantly improve the performance ceiling and robustness of each generation task field, and expands broader possibilities for achieving more realistic intelligent voice interaction and developing other voice-related products. For scenarios such as zero-shot TTS, it can ensure that the generated audio only has defects that are difficult for the human ear to perceive and can completely match the style of the given reference audio.

[0098] Figure 2 is a block diagram of a speech generating device provided according to an embodiment of the present disclosure. Figure 2 As shown, the speech generating device 200 comprises:

[0099] An acquisition module 201 is used to acquire multimodal audio data, wherein the multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data;

[0100] A quantization module 202, configured to uniformly quantize the multimodal audio data to obtain at least one quantization sequence corresponding to different modes;

[0101] The splicing module 203 is used to splice the at least one quantization sequence according to the target speech generation task to obtain a target quantization sequence, wherein the target speech generation task includes one or more of the following: a non-human synthesis TTA task, a music generation SVS task, a speech conversion VC task, a speech noise reduction SE task, and a speech synthesis TTS task;

[0102] The prediction module 204 is used to input the target quantization sequence and the conditional sequence into a pre-trained speech generation model to generate a predicted speech sequence.

[0103] Optionally, the quantization module 202 is used to:

[0104] For unimodal data of any mode in the multimodal audio data, input the unimodal data into a first encoder to obtain continuous latent features of each frame corresponding to the unimodal data;

[0105] Performing residual vector quantization RVQ on the latent features corresponds to a first quantization sequence of the unimodal data, wherein the first quantization sequence is to obtain N-bit discrete audio features, where N is greater than zero.

[0106] Optionally, the speech generating device 200 includes:

[0107] A reconstruction module, configured to input the first quantized sequence into a first decoder for reconstruction to obtain a first reconstructed audio corresponding to the single-modal data;

[0108] A discriminant module, used for inputting the first reconstructed audio and the unimodal data into a discriminator to obtain a discriminant result;

[0109] An updating module is used to update parameters of the first decoder and / or the first encoder according to the discrimination result.

[0110] Optionally, the condition sequence is determined according to the text data, the speech generation model includes at least a second encoder and a second decoder, and the prediction module 204 is used for:

[0111] Inputting the text data into a semantic recognition model to obtain the conditional sequence corresponding to the text data;

[0112] Inputting the condition sequence and the target quantization sequence into the second encoder for processing, thereby obtaining a first feature corresponding to the condition sequence and a second feature corresponding to the target quantization sequence;

[0113] Adding the first feature and the second feature in a unified dimension to obtain a fusion feature;

[0114] The fused features are input into the second decoder to generate a predicted speech sequence.

[0115] Optionally, the prediction module 204 is used to:

[0116] Inputting the fused features into the global encoding unit to obtain feature distribution information;

[0117] The feature distribution information and the fusion feature are input into the local encoding unit to obtain the predicted speech sequence.

[0118] Optionally, the second encoder includes a linear layer, and the prediction module 204 is used to:

[0119] The first feature and the second feature are input into the linear layer to obtain the fused feature.

[0120] Optionally, each element in each of the quantized sequences corresponds to a vocabulary identifier in a prediction vocabulary, and each speech generation task corresponds to a vocabulary identifier in the prediction vocabulary;

[0121] The splicing module 203 is used for:

[0122] The lexical identifier corresponding to the target speech generation task and the lexical identifier corresponding to each element in each of the quantization sequences are concatenated to obtain the target quantization sequence;

[0123] The target quantization sequence further includes a start tag and an end tag corresponding to the target speech generation task, and a start tag and an end tag corresponding to each of the quantization sequences.

[0124] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method or system, and will not be elaborated here.

[0125] It should be noted that the implementation principle of the aforementioned speech generation device 200 can refer to the implementation principle of the aforementioned speech generation method, and will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the speech generation device 200 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above speech generation device 200. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0126] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0127] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned speech generation method. Figure 3 As shown, Figure 3 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a speech generating device 200 , a memory 111 , a processor 112 and a communication unit 113 .

[0128] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The speech generating device 200 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the speech generation method stored in the memory 111, such as the software function module and computer program included in the speech generating device 200.

[0129] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the aforementioned speech generation method.

[0130] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A method for generating speech, It is characterized in that The method comprises: Acquire multimodal audio data, wherein the multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data; Performing unified quantization on the multimodal audio data to obtain at least one quantization sequence corresponding to different modes; According to a target speech generation task, the at least one quantization sequence is spliced ​​to obtain a target quantization sequence, wherein the target speech generation task includes one or more of the following: a non-human voice synthesis TTA task, a music generation SVS task, a speech conversion VC task, a speech noise reduction SE task, and a speech synthesis TTS task; Inputting the target quantization sequence and the conditional sequence into a pre-trained speech generation model to generate a predicted speech sequence; The step of uniformly quantizing the multimodal audio data to obtain at least one quantization sequence corresponding to different modes includes: For unimodal data of any mode in the multimodal audio data, input the unimodal data into a first encoder to obtain continuous latent features of each frame corresponding to the unimodal data; Inputting the latent feature into a residual vector quantization (RVQ) unit, corresponding to a first quantization sequence of the single-modal data, wherein the first quantization sequence is to obtain N-bit discrete audio features, where N is greater than; The method comprises: Inputting the first quantized sequence into a first decoder for reconstruction to obtain a first reconstructed audio corresponding to the single modal data; Inputting the first reconstructed audio and the unimodal data into a discriminator to obtain a discrimination result; According to the determination result, updating parameters of the first decoder and / or the first encoder; The conditional sequence is determined according to the text data, the speech generation model includes at least a second encoder and a second decoder, and the target quantization sequence and the conditional sequence are input into the pre-trained speech generation model to generate a predicted speech sequence, including: Inputting the text data into a semantic recognition model to obtain the conditional sequence corresponding to the text data; Inputting the condition sequence and the target quantization sequence into the second encoder for processing respectively, to obtain a first feature corresponding to the condition sequence and a second feature corresponding to the target quantization sequence; Adding the first feature and the second feature in a unified dimension to obtain a fusion feature; Inputting the fused features into the second decoder to generate a predicted speech sequence; The second encoder includes a global encoding unit and a local encoding unit, and the step of inputting the fusion feature into a second decoder to generate a predicted speech sequence includes: Inputting the fused features into the global encoding unit to obtain feature distribution information; Inputting the feature distribution information and the fusion feature into the local encoding unit to obtain the predicted speech sequence; The second encoder includes a linear layer, and the first feature and the second feature are uniformly added in dimension to obtain a fusion feature, including: Inputting the first feature and the second feature into the linear layer to obtain the fused feature; Each element in each of the quantized sequences corresponds to a vocabulary identifier in a prediction vocabulary, and each speech generation task corresponds to a vocabulary identifier in the prediction vocabulary; The step of splicing the at least one quantization sequence according to the target speech generation task to obtain a target quantization sequence includes: The lexical identifier corresponding to the target speech generation task and the lexical identifier corresponding to each element in each of the quantization sequences are concatenated to obtain the target quantization sequence; The target quantization sequence further includes a start tag and an end tag corresponding to the target speech generation task, and a start tag and an end tag corresponding to each of the quantization sequences.

2. A speech generating device, It is characterized in that The device comprises: An acquisition module, used for acquiring multimodal audio data, wherein the multimodal audio data includes one or more of the following: audio data, text data, midi data, and phoneme data; A quantization module, used to uniformly quantize the multimodal audio data to obtain at least one quantization sequence corresponding to different modes; A splicing module is used to splice the at least one quantization sequence according to a target speech generation task to obtain a target quantization sequence, wherein the target speech generation task includes one or more of the following: a non-human voice synthesis TTA task, a music generation SVS task, a speech conversion VC task, a speech noise reduction SE task, and a speech synthesis TTS task; A prediction module, used for inputting the target quantization sequence and the conditional sequence into a pre-trained speech generation model to generate a predicted speech sequence; The quantization module is specifically used for: For unimodal data of any mode in the multimodal audio data, the unimodal data is input into a first encoder to obtain continuous latent features of each frame corresponding to the unimodal data; the latent features are input into a residual vector quantization (RVQ) unit, corresponding to a first quantization sequence of the unimodal data, wherein the first quantization sequence is to obtain N-bit discrete audio features, where N is greater than zero; The acquisition module also: Inputting the first quantized sequence into a first decoder for reconstruction to obtain a first reconstructed audio corresponding to the unimodal data; inputting the first reconstructed audio and the unimodal data into a discriminator to obtain a discrimination result; and updating parameters of the first decoder and / or the first encoder according to the discrimination result; The condition sequence is determined according to the text data, the speech generation model comprises at least a second encoder and a second decoder, and the prediction module is specifically used for: Input the text data into a semantic recognition model to obtain the conditional sequence corresponding to the text data; input the conditional sequence and the target quantization sequence into the second encoder for processing respectively to obtain a first feature corresponding to the conditional sequence and a second feature corresponding to the target quantization sequence; perform dimension-wise addition of the first feature and the second feature to obtain a fused feature; input the fused feature into the second decoder to generate a predicted speech sequence; The second encoder includes a global encoding unit and a local encoding unit, and the prediction module is further used for: Inputting the fused features into the global encoding unit to obtain feature distribution information; inputting the feature distribution information and the fused features into the local encoding unit to obtain the predicted speech sequence; The second encoder includes a linear layer, and the prediction module is further configured to: Inputting the first feature and the second feature into the linear layer to obtain the fused feature; Each element in each of the quantized sequences corresponds to a vocabulary identifier in a prediction vocabulary, and each speech generation task corresponds to a vocabulary identifier in the prediction vocabulary; The splicing module is specifically used for: The lexical identifier corresponding to the target speech generation task and the lexical identifier corresponding to each element in each of the quantization sequences are concatenated to obtain the target quantization sequence; The target quantization sequence further includes a start tag and an end tag corresponding to the target speech generation task, and a start tag and an end tag corresponding to each of the quantization sequences.

3. A computer device, It is characterized in that The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the speech generation method according to claim 1.

4. A readable storage medium, It is characterized in that The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the speech generation method according to claim 1.