Zero sample speech synthesis method and device based on autoregressive large language model

Abstract

The invention provides a zero-sample speech synthesis method and device based on an autoregressive large language model, and the method comprises the steps: inputting an audio coding feature of a reference audio, a text coding feature of a reference text, and a target text coding feature of a target text into a pre-trained large language model, obtaining a target sequence output by the large language model; inputting the target sequence into a pre-trained diffusion model to obtain a target Mel spectrum corresponding to the target sequence output by the diffusion model; and inputting the target Mel spectrum into a pre-trained vocoder to obtain a target audio corresponding to the target Mel spectrum output by the vocoder, and determining the target audio as a speech synthesis result corresponding to the reference audio and the target text. Therefore, a user only needs to add a small amount of reference audio of the target pronunciation object in a speech synthesis stage, and does not need to use speech data of the target pronunciation object to repeatedly train an original speech synthesis model, so that the zero-sample speech synthesis capability is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a zero-sample speech synthesis method and device based on an autoregressive large language model. Background Art

[0002] The existing speech synthesis model usually first uses the speech data of a pronunciation object (including audio and text description information corresponding to the audio) to train the speech synthesis model, so that the trained speech synthesis model can generate synthesized speech that matches the input text information according to the pronunciation characteristics of the above pronunciation object (such as timbre, intonation, reading emotion, etc.). However, when the user wants to obtain synthesized speech generated according to the pronunciation characteristics of other pronunciation objects, it is necessary to perform a new round of training on the above speech synthesis model based on the speech data of the newly added other pronunciation objects, which makes it impossible for the user to flexibly and freely select the pronunciation object corresponding to the speech to be synthesized, and when a new selectable pronunciation object is added, the model training time will also increase, resulting in a decrease in speech synthesis efficiency. Summary of the invention

[0003] In view of this, the present application provides a zero-sample speech synthesis method and device based on an autoregressive large language model. By constructing a zero-sample speech synthesis model, the user only needs to add a small amount of reference audio of the target pronunciation object in the speech synthesis stage, and there is no need to repeatedly train the original speech synthesis model using the speech data of the target pronunciation object, which effectively improves the zero-sample speech synthesis capability, is conducive to improving the flexibility of selecting pronunciation objects during speech synthesis and reducing the model training cost of the speech synthesis model.

[0004] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0005] In a first aspect, an embodiment of the present application provides a zero-sample speech synthesis method based on an autoregressive large language model, the zero-sample speech synthesis method comprising:

[0006] Inputting the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; wherein the reference text is text information that matches the reference audio; the audio features corresponding to the target sequence match the reference audio, and the semantic features corresponding to the target sequence match the reference text;

[0007] Inputting the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model;

[0008] The target Mel spectrum is input into a pre-trained vocoder to obtain a target audio corresponding to the target Mel spectrum output by the vocoder, and the target audio is determined as a speech synthesis result corresponding to the reference audio and the target text.

[0009] In a second aspect, an embodiment of the present application provides a zero-sample speech synthesis device based on an autoregressive large language model, the zero-sample speech synthesis device comprising:

[0010] A first processing module is used to input the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; wherein the reference text is text information that matches the reference audio; the audio features corresponding to the target sequence match the reference audio, and the semantic features corresponding to the target sequence match the reference text;

[0011] A second processing module is used to input the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model;

[0012] The third processing module is used to input the target Mel spectrum into a pre-trained vocoder, obtain the target audio corresponding to the target Mel spectrum output by the vocoder, and determine the target audio as the speech synthesis result corresponding to the reference audio and the target text.

[0013] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned zero-sample speech synthesis method based on an autoregressive large language model when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned zero-sample speech synthesis method based on an autoregressive large language model are executed.

[0015] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0016] The embodiment of the present application provides a zero-sample speech synthesis method and device based on an autoregressive large language model, which inputs the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; inputs the target sequence into a pre-trained diffusion model to obtain a target Mel spectrum corresponding to the target sequence output by the diffusion model; inputs the target Mel spectrum into a pre-trained vocoder to obtain a target audio corresponding to the target Mel spectrum output by the vocoder, and determines the target audio as the speech synthesis result corresponding to the reference audio and the target text. In this way, by constructing a zero-sample speech synthesis model, the user only needs to add a small amount of reference audio of the target pronunciation object in the speech synthesis stage, and there is no need to use the speech data of the target pronunciation object to repeatedly train the original speech synthesis model, which effectively improves the zero-sample speech synthesis capability, is conducive to improving the flexibility of selecting the pronunciation object during speech synthesis, and reducing the model training cost of the speech synthesis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic flow chart of a zero-sample speech synthesis method based on an autoregressive large language model provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic flow chart of a method for obtaining audio coding features of reference audio provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic flow chart of a method for obtaining text encoding features corresponding to a reference text and a target text provided in an embodiment of the present application is shown;

[0021] Figure 4 A flow chart of a model training method for training a large language model provided in an embodiment of the present application is shown;

[0022] Figure 5 A schematic diagram of the structure of a zero-sample speech synthesis device based on an autoregressive large language model provided in an embodiment of the present application is shown;

[0023] Figure 6 A schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0025] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0026] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0027] In one embodiment of the present application, a zero-sample speech synthesis method based on an autoregressive large language model can be run on a terminal device or a server. The terminal device can be a local terminal device. When the zero-sample speech synthesis method is run on a server, the zero-sample speech synthesis method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device (i.e., a terminal device).

[0028] To facilitate understanding of the embodiments of the present application, a zero-sample speech synthesis method and device based on an autoregressive large language model provided in the embodiments of the present application are introduced in detail below.

[0029] Reference Figure 1 As shown, Figure 1 A flowchart of a zero-sample speech synthesis method based on an autoregressive large language model provided in an embodiment of the present application is shown, wherein the zero-sample speech synthesis method includes steps S101-S103; specifically:

[0030] S101, inputting audio coding features of reference audio, text coding features of reference text, and target text coding features of target text into a pre-trained large language model to obtain a target sequence output by the large language model.

[0031] Here, the reference audio can be the voice audio of the target pronunciation object selected by the user, and the reference text is the text information matching the reference audio. For example, if the reference audio is the "abcd" voice information read by the target pronunciation object, the reference text can be the "abcd" information in text form.

[0032] Here, the target text represents the text information that the user specifies to be read aloud according to the voice of the target pronunciation object. For example, if the user specifies that the target pronunciation object needs to read a voice information of "efgh", the above target text can be the "efgh" information in text form.

[0033] It should be noted that the specific text content of the target text can be determined according to the user's current speech synthesis requirements, and the specific text content of the target text has no connection with the audio content of the reference audio and the text content of the reference text.

[0034] Here, the large language model may be a pre-trained LLM (Large Language Model) model.

[0035] Specifically, the reference audio, reference text and target text can be pre-processed into feature data of a vector type that can be recognized by a large language model to obtain audio coding features of the reference audio, text coding features of the reference text, and target text coding features of the target text; then, the above audio coding features, the above text coding features and the above target text coding features are input as prompt words into a pre-trained large language model, and the large language model can output a target sequence whose audio features match the reference audio and whose semantic features match the reference text (equivalent to the large language model predicting the speech data of the target text to be read according to the pronunciation features of the target pronunciation object in the reference audio, and the vector representation result corresponding to the above speech data outputted is the target sequence).

[0036] It should be noted that since the pronunciation of the target pronunciation object in the reference audio corresponds to the text in the reference text, when the above-mentioned target sequence is output through the large language model, in addition to the audio coding features of the above-mentioned reference audio and the target text coding features of the above-mentioned target text, it is also necessary to input the text coding features of the reference text. In the process of the large language model outputting the above-mentioned target sequence, the large language model is helped to learn and understand the mapping relationship between the audio features in the reference audio (i.e., the above-mentioned audio coding features) and the text features of the text information corresponding to it (i.e., the text coding features of the above-mentioned reference text), so as to improve the accuracy of the output target sequence.

[0037] S102, inputting the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model.

[0038] Here, the diffusion model may be a pre-trained Diffusion Model model. By inputting the target sequence output by the large language model into the pre-trained diffusion model, a target Mel spectrum (ie, target Mel spectrum) corresponding to the target sequence output by the diffusion model may be obtained.

[0039] Specifically, when training a diffusion model, the training data set may include: multiple audio feature sequences and a real Mel spectrum corresponding to each audio feature sequence; wherein, for each audio feature sequence, the audio feature sequence is input into the initial model of the above-mentioned diffusion model, and the Mel spectrum prediction result corresponding to the audio feature sequence output by the above-mentioned initial model can be obtained. According to the loss between the Mel spectrum prediction result and the real Mel spectrum corresponding to the audio feature sequence, the above-mentioned initial model can be trained until the above-mentioned initial model reaches convergence, so as to obtain the trained above-mentioned diffusion model.

[0040] It should be noted that, similar to the above-mentioned target sequence, the audio feature sequence in the above-mentioned training data set can be obtained in the same manner as the above-mentioned target sequence; for example, the audio encoding features of the sample audio, the text encoding features of the text information matching the sample audio, and the text encoding features of the original text can be input into a pre-trained large language model, and the above-mentioned audio feature sequence whose audio features match the above-mentioned sample audio and whose semantic features match the above-mentioned original text can be obtained through the output of the large language model.

[0041] S103, inputting the target Mel-spectrogram into a pre-trained vocoder to obtain a target audio corresponding to the target Mel-spectrogram output by the vocoder, and determining the target audio as a speech synthesis result corresponding to the reference audio and the target text.

[0042] Here, the target mel spectrum is input into a pre-trained vocoder, which can reconstruct the input target mel spectrum into an audio waveform and output a target audio corresponding to the target mel spectrum. At this time, the target audio is equivalent to the speech data of reading the target text according to the pronunciation characteristics of the target pronunciation object in the reference audio (i.e., the timbre, intonation, reading emotion, etc. of the target pronunciation object), that is, the target audio is the speech synthesis result corresponding to the reference audio and the target text.

[0043] It should be noted that the embodiments of the present application do not impose any limitation on the specific model and specific model structure of the above-mentioned vocoder.

[0044] The specific implementation process of each of the above steps in the embodiment of the present application is described in detail below:

[0045] Regarding the specific implementation process of the above step S101, in an optional implementation mode, Figure 2 A flow chart of a method for obtaining audio coding features of reference audio provided in an embodiment of the present application is shown. Figure 2 As shown, before executing the above step S101, the method includes steps S201-S202, specifically:

[0046] S201, discretize the reference audio to obtain discrete audio features of the reference audio.

[0047] Here, the encoder and quantizer in a pre-trained neural network can be used to discretize the reference audio; wherein the neural network is composed of an encoder, a quantizer and a decoder. When training the neural network, the original speech can be input into the encoder, and the encoder can encode and output the input original speech as an intermediate vector v of a preset fixed dimension. The intermediate vector v output by the encoder is input into the quantizer, and the quantizer performs vector quantization or residual vector quantization on the input intermediate vector v to output a discrete audio feature y corresponding to the intermediate vector v. The discrete audio feature y output by the quantizer is input into the decoder, and the discrete audio feature y is decoded by the decoder to output a speech restoration result of the original speech. According to the loss between the speech restoration result and the original speech, the neural network can be trained until the neural network converges.

[0048] Specifically, as an optional embodiment, the encoder and quantizer trained in the neural network may be used to perform the step S201 according to the method described in the following steps a1-a2:

[0049] Step a1: input the reference audio into an encoder to obtain an intermediate feature vector of the reference audio output by the encoder.

[0050] Here, similar to the above-mentioned original speech, the reference audio is input into the encoder, and the encoder can encode and output the input reference audio into an intermediate feature vector of a preset fixed dimension (ie, the intermediate feature vector of the reference audio).

[0051] Step a2: input the intermediate feature vector into a quantizer, and perform vector quantization processing or residual vector quantization processing on the intermediate feature vector through the quantizer to obtain discrete audio features output by the quantizer as discrete audio features of the reference audio.

[0052] Specifically, taking the codebook W in the quantizer as an example, the codebook W has n codebook vectors w1, w2, ...w with a fixed order. n When the input intermediate feature vector is vector quantized by the quantizer, the quantizer retrieves the codebook vector w with the highest similarity to the input intermediate feature vector from the codebook W. i , and use the retrieved codebook vector w i The intermediate feature vector is replaced and output as discrete audio features; wherein the discrete audio features are composed of a set of discrete tokens (features of unit length).

[0053] Specifically, when the input intermediate feature vector is subjected to residual vector quantization processing through the quantizer, the input intermediate feature vector can be quantized in stages, and the specific number of quantization stages involved in this process can be configured according to actual needs, and vector quantization processing is performed once in each quantization stage; wherein, the input intermediate feature vector is subjected to the first vector quantization processing, and the codebook vector w1 with the highest similarity to the intermediate feature vector can be retrieved from the first-layer codebook, and then the residual R1 between the intermediate feature vector and the codebook vector w1 is calculated, and then the residual R1 can be subjected to the second vector quantization processing, and the codebook vector w2 with the highest similarity to the residual R1 can be retrieved from the second-layer codebook, and the residual R2 between the residual R1 and the codebook vector w2 is calculated, and so on until the specific number of quantization times pre-configured before is reached.

[0054] It should be noted that, considering that the residual vector quantization processing only needs to approximately represent the input vector of this layer in the quantization process of each layer, a smaller number of codebooks can be used to obtain a more accurate vector representation, and a higher quantization accuracy can be obtained at a lower bit rate, providing a higher compression ratio and better compression performance. Therefore, in practical applications, the residual vector quantization processing can be preferentially used to process the above-mentioned intermediate feature vectors.

[0055] S202: Input the discrete audio features into an audio encoder to obtain audio coding features output by the audio encoder as audio coding features of the reference audio.

[0056] Here, the discrete audio features output from the quantizer are input into the audio encoder, and the discrete audio features are encoded by the audio encoder to output the audio coding features of the reference audio.

[0057] Regarding the specific implementation process of the above step S101, in an optional implementation mode, Figure 3 A flow chart of a method for obtaining text encoding features corresponding to a reference text and a target text provided in an embodiment of the present application is shown. Figure 3 As shown, before executing the above step S101, the method includes steps S301-S303, specifically:

[0058] S301, discretizing the reference text and the target text respectively to obtain discrete text features of the reference text and target discrete text features of the target text.

[0059] Here, the reference text and the target text are both text information, so the same discretization processing method can be used (as long as it is a discretization processing method applicable to text information), and the reference text and the target text are discretized separately to obtain the discrete text features of the reference text and the target discrete text features of the target text.

[0060] It should be noted that in step S301, the discretization processing method that can be used for the reference text and the target text is not unique. For example, the reference text and the target text can be separately segmented, or the reference text and the target text can be classified into different categories or labels for discretization processing, etc. The embodiment of the present application does not impose any limitation on the specific discretization processing method used in step S301.

[0061] S302: Input the discrete text features into a text encoder to obtain text encoding features output by the text encoder as text encoding features of the reference text.

[0062] Here, after the reference text is discretized to obtain discrete text features of the reference text, the discrete text features can be input into a text encoder, and the discrete text features are encoded by the text encoder to output text encoding features of the reference text.

[0063] S303: Input the target discrete text feature into a text encoder to obtain a target text encoding feature output by the text encoder as a target text encoding feature of the target text.

[0064] Here, after the target text is discretized to obtain the target discrete text features of the target text, the target discrete text features can be input into a text encoder, and the target discrete text features are encoded by the text encoder to output the target text encoded features.

[0065] For the large language model in step S101 above, in an optional implementation, Figure 4 A flow chart of a model training method for training a large language model provided in an embodiment of the present application is shown, such as Figure 4 As shown, before executing the above step S101, the model training method includes steps S401-S403, specifically:

[0066] S401, collecting audio sample data of multiple different sound-making objects and text sample information matching the audio sample data.

[0067] Here, voice data of multiple different sound-making objects reading different text information can be collected in a variety of different sound-making scenarios (such as in a quiet scene with low decibels, in a noisy scene with relatively more noise, etc.), wherein the voice data read by each sound-making object is the above-mentioned audio sample data, and the text information read by each sound-making object is the text sample information matching the voice data of the sound-making object (i.e., the text sample information matching the above-mentioned audio sample data).

[0068] Specifically, when collecting the above-mentioned voice data, each collected voice data can be uniformly converted to the same fixed sampling rate, and after certain preprocessing (such as loudness normalization, conversion to a single channel, segmentation processing of long voice data, etc.), each preprocessed voice data is obtained as the above-mentioned audio sample data.

[0069] It should be noted that in order to enrich the pronunciation characteristics of the sound-speaking subjects in the audio sample data, voice data of sound-speaking subjects of different age groups under a variety of different reading emotions can be collected. In addition, multiple sound-speaking subjects with a gender ratio that is as balanced as possible can be selected.

[0070] S402, for each of the audio sample data, input the audio coding features of the audio sample data, the text coding features of the text sample information matching the audio sample data, and the text coding features of the text to be processed into the original model of the large language model to obtain the target sample sequence output by the original model.

[0071] Here, the text to be processed represents the text sample information that matches the audio sample data; that is, for each audio sample data, the text sample information that matches the audio sample data can be directly copied as the above-mentioned text to be processed, or the text encoding features of the text sample information that matches the audio sample data can be directly copied as the text encoding features of the above-mentioned text to be processed.

[0072] For each audio sample data, the original model can learn the pronunciation features of different characters in the text sample information according to the audio coding features of the input audio sample data and the text coding features (i.e., the text coding features of the text sample information matching the audio sample data), and then output a character distribution sequence containing pronunciation features as the target sample sequence according to the distribution positions of different characters in the text to be processed; wherein, when outputting the target sample sequence, the original model can also predict the next character containing pronunciation features (i.e., the next unit length token) based on the previous character containing pronunciation features that has been generated (also equivalent to a unit length token), until a complete target sample sequence is generated (equivalent to the target sample sequence being a sequence composed of the above-predicted unit length tokens).

[0073] S403, adjusting the model parameters of the original model according to the audio feature loss between the target sample sequence and the audio sample data and the semantic feature loss between the target sample sequence and the text to be processed, and obtaining the original model including the adjusted parameters as the trained large language model.

[0074] Here, as an optional embodiment, a preset model loss function (such as a cross entropy loss function) can be directly used to calculate the audio feature loss between the target sample sequence and the audio sample data (i.e., the feature loss between the target sample sequence and the audio coding features of the audio sample data) and the semantic feature loss between the target sample sequence and the text to be processed (i.e., the feature loss between the target sample sequence and the text coding features of the text to be processed), and the sum of the above audio feature loss and the above semantic feature loss is used as the overall model loss of the original model. The original model is trained until the original model converges (equivalent to continuously adjusting the model parameters of the original model until the above overall model loss is minimized), and the converged original model is obtained as a trained large language model (i.e., the original model including the adjusted parameters is obtained as a trained large language model).

[0075] Here, as another optional embodiment, a multi-head cross-attention module can be connected behind the original model. The multi-head cross-attention module can self-learn the joint information between the target sample sequence, the text encoding features of the text to be processed, and the audio encoding features of the audio sample data, thereby supervising the alignment of the target sample sequence with the audio sample data in the pronunciation feature dimension and the alignment of the target sample sequence with the text to be processed in the semantic feature dimension, thereby improving the model training accuracy of the large language model.

[0076] Specifically, after the multi-head cross attention module is connected, the audio feature loss and the semantic feature loss can be determined according to the method described in steps b1-b3 below:

[0077] Step b1, input the audio coding features of the audio sample data, the text coding features of the text to be processed and the target sample sequence into a multi-head cross-attention module, and output the alignment difference between the target sample sequence and the audio coding features of the audio sample data and the alignment difference between the target sample sequence and the text coding features of the text to be processed respectively through the multi-head cross-attention module.

[0078] Here, in the multi-head cross-attention module, a token of unit length can be used as the comparison unit. By comparing the feature gap between the target sample sequence and the above-mentioned audio coding features on each token, the cumulative feature gap is obtained as the alignment gap between the target sample sequence and the audio coding features of the audio sample data.

[0079] Similarly, a token of unit length can be used as the comparison unit, and the feature gap between the target sample sequence and the text encoding feature of the text to be processed at each token can be compared to obtain the cumulative feature gap as the alignment gap between the target sample sequence and the text encoding feature of the text to be processed.

[0080] Step b2: Determine the alignment difference between the target sample sequence and the audio coding features of the audio sample data as the audio feature loss between the target sample sequence and the audio sample data.

[0081] Here, in combination with the specific implementation of step b1 above, the alignment difference between the target sample sequence and the audio coding features of the audio sample data may be taken as the audio feature loss between the target sample sequence and the audio sample data.

[0082] Step b3: determining the alignment difference between the target sample sequence and the text encoding features of the text to be processed as the semantic feature loss between the target sample sequence and the text to be processed.

[0083] Here, in combination with the specific implementation of step b1 above, the alignment difference between the target sample sequence and the text encoding features of the text to be processed can be used as the semantic feature loss between the target sample sequence and the text to be processed.

[0084] For the vocoder in the above step S103, in an embodiment of the present application, a HiFiGAN vocoder can be selected. In addition, based on the HiFiGAN vocoder, the one-dimensional convolution in the generator of the HiFiGAN vocoder can be replaced with a one-dimensional depth-separable convolution, and the replaced HiFiGAN vocoder is used as the vocoder in step S103.

[0085] It should be noted that the HiFiGAN vocoder uses a generative adversarial network as the basic generation model, which includes a generator and a discriminator (which can be divided into a multi-scale discriminator and a multi-cycle discriminator). The generator of the HiFiGAN vocoder mainly consists of two parts, one is the upsampling structure (specifically composed of one-dimensional convolution); the other is the multi-receptive field fusion module (mainly responsible for optimizing the sampling points obtained by upsampling, specifically composed of a residual network). In order to further reduce the number of parameters and improve the reasoning speed, the embodiment of the present application replaces the above-mentioned one-dimensional convolution in the generator with a one-dimensional deep separable convolution. Compared with the standard one-dimensional convolution, which jointly maps the time correlation and channel correlation, the replaced one-dimensional deep separable convolution can map the time correlation and channel correlation separately. From the process of extracting the Mel spectrum from the speech waveform, the replaced one-dimensional deep separable convolution can first segment the speech, and then calculate and extract a frame of Mel spectrum for each segment of speech. Therefore, the embodiment of the present application uses a one-dimensional deep separable convolution to separately learn the time correlation and channel correlation in the Mel spectrum. This improved HiFiGAN vocoder is more helpful for the vocoder to restore the speech waveform using the Mel spectrum, and relatively reduces the learning burden of the generative adversarial network in the vocoder, which helps to quickly learn the generative adversarial network.

[0086] Based on the above-mentioned zero-sample speech generation method based on the autoregressive large language model of the embodiment of the present application, the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text are input into the pre-trained large language model to obtain the target sequence output by the large language model; the target sequence is input into the pre-trained diffusion model to obtain the target Mel spectrum corresponding to the target sequence output by the diffusion model; the target Mel spectrum is input into the pre-trained vocoder to obtain the target audio corresponding to the target Mel spectrum output by the vocoder, and the target audio is determined as the speech synthesis result corresponding to the reference audio and the target text. In this way, by constructing a zero-sample speech synthesis model, the user only needs to add a small amount of reference audio of the target pronunciation object in the speech synthesis stage, and there is no need to use the speech data of the target pronunciation object to repeatedly train the original speech synthesis model, which effectively improves the zero-sample speech synthesis capability, is conducive to improving the flexibility of selecting the pronunciation object during speech synthesis and reducing the model training cost of the speech synthesis model.

[0087] Based on the same inventive concept, the present application also provides a zero-sample speech synthesis device corresponding to the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model. Since the principle of solving the problem by the zero-sample speech synthesis device in the embodiment of the present application is similar to the above-mentioned zero-sample speech synthesis method in the embodiment of the present application, the implementation of the zero-sample speech synthesis device can refer to the implementation of the above-mentioned zero-sample speech synthesis method, and the repeated parts will not be repeated.

[0088] Reference Figure 5 As shown, Figure 5 A schematic diagram of the structure of a zero-sample speech synthesis device based on an autoregressive large language model provided in an embodiment of the present application is shown, wherein the zero-sample speech synthesis device includes:

[0089] The first processing module 501 is used to input the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; wherein the reference text is text information matching the reference audio; the audio features corresponding to the target sequence match the reference audio, and the semantic features corresponding to the target sequence match the reference text;

[0090] The second processing module 502 is used to input the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model;

[0091] The third processing module 503 is used to input the target Mel spectrum into a pre-trained vocoder, obtain the target audio corresponding to the target Mel spectrum output by the vocoder, and determine the target audio as the speech synthesis result corresponding to the reference audio and the target text.

[0092] In an optional implementation, the first processing module 501 is configured to obtain the audio coding feature of the reference audio by the following method:

[0093] Performing discretization processing on the reference audio to obtain discrete audio features of the reference audio;

[0094] The discrete audio features are input into an audio encoder, and audio coding features output by the audio encoder are obtained as audio coding features of the reference audio.

[0095] In an optional implementation, when the reference audio is discretized to obtain discrete audio features of the reference audio, the first processing module 501 is configured to:

[0096] Inputting the reference audio into an encoder to obtain an intermediate feature vector of the reference audio output by the encoder;

[0097] The intermediate feature vector is input into a quantizer, and the intermediate feature vector is subjected to vector quantization processing or residual vector quantization processing by the quantizer to obtain discrete audio features output by the quantizer as discrete audio features of the reference audio.

[0098] In an optional implementation, the first processing module 501 is used to obtain the text encoding features of the reference text and the target text encoding features of the target text by the following method:

[0099] Discretizing the reference text and the target text respectively to obtain discrete text features of the reference text and target discrete text features of the target text;

[0100] Inputting the discrete text features into a text encoder to obtain text encoding features output by the text encoder as text encoding features of the reference text;

[0101] The target discrete text features are input into a text encoder, and the target text encoding features output by the text encoder are obtained as the target text encoding features of the target text.

[0102] In an optional implementation, the first processing module 501 is used to train the large language model by the following method:

[0103] Collecting audio sample data of multiple different sound-making objects and text sample information matching the audio sample data;

[0104] For each of the audio sample data, the audio coding features of the audio sample data, the text coding features of the text sample information matching the audio sample data, and the text coding features of the text to be processed are input into the original model of the large language model to obtain a target sample sequence output by the original model; wherein the text to be processed represents the text sample information matching the audio sample data;

[0105] According to the audio feature loss between the target sample sequence and the audio sample data and the semantic feature loss between the target sample sequence and the text to be processed, the model parameters of the original model are adjusted to obtain the original model including the adjusted parameters as the trained large language model.

[0106] In an optional implementation, the first processing module 501 is used to determine the audio feature loss between the target sample sequence and the audio sample data and the semantic feature loss between the target sample sequence and the text to be processed by the following method:

[0107] Input the audio coding features of the audio sample data, the text coding features of the text to be processed, and the target sample sequence into a multi-head cross attention module, and output the alignment difference between the target sample sequence and the audio coding features of the audio sample data and the alignment difference between the target sample sequence and the text coding features of the text to be processed respectively through the multi-head cross attention module;

[0108] Determine the alignment difference between the target sample sequence and the audio coding features of the audio sample data as the audio feature loss between the target sample sequence and the audio sample data;

[0109] The alignment difference between the target sample sequence and the text encoding features of the text to be processed is determined as the semantic feature loss between the target sample sequence and the text to be processed.

[0110] In an optional embodiment, the vocoder represents a HiFiGAN vocoder after replacing the one-dimensional convolution in the generator with a one-dimensional depthwise separable convolution.

[0111] Based on the same inventive concept, the present application also provides an electronic device corresponding to the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model, and the repeated parts will not be repeated.

[0112] like Figure 6 As shown, an embodiment of the present application provides an electronic device 600 for executing the zero-sample speech synthesis method based on an autoregressive large language model in the present application. The device includes a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602, wherein the memory 601 and the processor 602 are communicatively connected via a bus, and the processor 602 implements the steps of the zero-sample speech synthesis method based on an autoregressive large language model when executing the computer program.

[0113] Specifically, the above-mentioned memory 601 and processor 602 can be general-purpose memory and processor, which are not specifically limited here. When the processor 602 runs the computer program stored in the memory 601, it can execute the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model.

[0114] Corresponding to the speech synthesis method in the present application, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned zero-sample speech synthesis method based on the autoregressive large language model are executed.

[0115] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned speech synthesis method can be executed.

[0116] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0119] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0120] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A zero-shot speech synthesis method based on an autoregressive large language model, characterized in that: The zero-sample speech synthesis method comprises: Inputting the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; wherein the reference text is text information that matches the reference audio; the audio features corresponding to the target sequence match the reference audio, and the semantic features corresponding to the target sequence match the reference text; Inputting the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model; The target Mel spectrum is input into a pre-trained vocoder to obtain a target audio corresponding to the target Mel spectrum output by the vocoder, and the target audio is determined as a speech synthesis result corresponding to the reference audio and the target text.

2. The zero-sample speech synthesis method according to claim 1, characterized in that: The audio coding feature of the reference audio is obtained by the following method: Performing discretization processing on the reference audio to obtain discrete audio features of the reference audio; The discrete audio features are input into an audio encoder, and audio coding features output by the audio encoder are obtained as audio coding features of the reference audio.

3. The zero-sample speech synthesis method according to claim 2, characterized in that: The discretizing the reference audio to obtain discrete audio features of the reference audio includes: Inputting the reference audio into an encoder to obtain an intermediate feature vector of the reference audio output by the encoder; The intermediate feature vector is input into a quantizer, and the intermediate feature vector is subjected to vector quantization processing or residual vector quantization processing by the quantizer to obtain discrete audio features output by the quantizer as discrete audio features of the reference audio.

4. The zero-sample speech synthesis method according to claim 1, characterized in that: The text encoding features of the reference text and the target text encoding features of the target text are obtained by the following method: Discretizing the reference text and the target text respectively to obtain discrete text features of the reference text and target discrete text features of the target text; Inputting the discrete text features into a text encoder to obtain text encoding features output by the text encoder as text encoding features of the reference text; The target discrete text features are input into a text encoder, and the target text encoding features output by the text encoder are obtained as the target text encoding features of the target text.

5. The zero-sample speech synthesis method according to claim 1, characterized in that: The large language model is obtained by training using the following method: Collecting audio sample data of multiple different sound-making objects and text sample information matching the audio sample data; For each of the audio sample data, the audio coding features of the audio sample data, the text coding features of the text sample information matching the audio sample data, and the text coding features of the text to be processed are input into the original model of the large language model to obtain a target sample sequence output by the original model; wherein the text to be processed represents the text sample information matching the audio sample data; According to the audio feature loss between the target sample sequence and the audio sample data and the semantic feature loss between the target sample sequence and the text to be processed, the model parameters of the original model are adjusted to obtain the original model including the adjusted parameters as the trained large language model.

6. The zero-sample speech synthesis method according to claim 5, characterized in that: The audio feature loss between the target sample sequence and the audio sample data and the semantic feature loss between the target sample sequence and the text to be processed are determined by the following method: Input the audio coding features of the audio sample data, the text coding features of the text to be processed, and the target sample sequence into a multi-head cross attention module, and output the alignment difference between the target sample sequence and the audio coding features of the audio sample data and the alignment difference between the target sample sequence and the text coding features of the text to be processed respectively through the multi-head cross attention module; Determine the alignment difference between the target sample sequence and the audio coding features of the audio sample data as the audio feature loss between the target sample sequence and the audio sample data; The alignment difference between the target sample sequence and the text encoding features of the text to be processed is determined as the semantic feature loss between the target sample sequence and the text to be processed.

7. The zero-sample speech synthesis method according to claim 1, characterized in that: The vocoder representation is a HiFiGAN vocoder after replacing the one-dimensional convolution in the generator with a one-dimensional depthwise separable convolution.

8. A zero-sample speech synthesis device based on an autoregressive large language model, characterized in that: The zero-sample speech synthesis device comprises: A first processing module is used to input the audio coding features of the reference audio, the text coding features of the reference text, and the target text coding features of the target text into a pre-trained large language model to obtain a target sequence output by the large language model; wherein the reference text is text information that matches the reference audio; the audio features corresponding to the target sequence match the reference audio, and the semantic features corresponding to the target sequence match the reference text; A second processing module is used to input the target sequence into a pre-trained diffusion model to obtain a target mel spectrum corresponding to the target sequence output by the diffusion model; The third processing module is used to input the target Mel spectrum into a pre-trained vocoder, obtain the target audio corresponding to the target Mel spectrum output by the vocoder, and determine the target audio as the speech synthesis result corresponding to the reference audio and the target text.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the zero-sample speech synthesis method based on the autoregressive large language model are performed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the zero-sample speech synthesis method based on an autoregressive large language model as described in any one of claims 1 to 7.

Images