Method and apparatus for generating sound, storage medium, electronic device

Abstract

The present invention provides a voice generation method, an apparatus, a storage medium, and an electronic device. The method includes: obtaining a voice feature vector of a voice to be processed, and inputting the voice feature vector into a voice generation model to obtain a language unit vector (S110); obtaining a text feature vector, and determining a feature vector to be processed based on the text feature vector and the language unit vector (S120); inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target voice corresponding to the voice to be processed or the text feature vector (S130).

Description

Related applications

[0001] This disclosure claims priority to the Chinese patent application filed on 27 May 2022, with application number 202210593870.9, titled "Speech Generation Method and Apparatus, Storage Medium, Electronic Device," and all contents of said Chinese patent application are incorporated herein by reference. [Technical Field]

[0002] This disclosure relates to the field of speech processing technology, and more particularly to speech generation methods and speech generation devices, computer-readable storage media and electronic devices. [Background technology]

[0003] In recent years, with the rapid development of deep learning, speech synthesis technology (Text to Speech, TTS) has made remarkable progress. Simultaneously, the development of various deep learning technologies has also led to rapid advancements in voice conversion (VC). However, both TTS and VC models require a large amount of recorded speech data (more than ten hours) to achieve the desired results. However, recording speech is extremely expensive and cumbersome. Therefore, guaranteeing effective speech synthesis and timbre conversion even with limited speaker data has become a hot topic in research. This research is called speaker adaptation or speaker cloning.

[0004] Currently, some research is being conducted that combines speech synthesis technology and speech timbre conversion. For example, by encoding different input source content using different encoders and then decoding it using the same decoder, it is possible to process two tasks simultaneously. However, in terms of effectiveness, the efficiency of TTS is always reduced, or, if different encoders are used to encode different input source content, the model training becomes complex and requires many loss functions and hyperparameters. Therefore, research combining speech synthesis technology and speech timbre conversion is not only complex in its training method, but it also fails to handle speech synthesis and speech timbre conversion well, and thus cannot simultaneously improve the performance of both tasks.

[0005] In light of this, it is necessary to develop new speech generation methods and devices in this field.

[0006] Furthermore, the information partially disclosed in the above background technology is used solely to enhance understanding of the background of this disclosure and may include information that does not constitute prior art known to those skilled in the art. [Overview of the Initiative]

[0007] The purpose of this disclosure is to provide a speech generation method, a speech generation apparatus, a computer-readable storage medium, and an electronic device that can overcome, at least to some extent, the technical problems of low fusion effect between speech synthesis technology and speech timbre conversion due to limitations of related technologies, and the difficulty in simultaneously improving the performance of the two tasks.

[0008] Other features and benefits of this disclosure will become apparent from the detailed description below or will be partially learned from the implementation of this disclosure.

[0009] According to a first embodiment of the present invention, a speech generation method is provided, and the speech generation method is The process involves obtaining the speech feature vector of the audio to be processed, inputting the speech feature vector into a speech generation model to obtain a language unit vector, and The process involves obtaining a text feature vector, determining the feature vector to be processed based on the text feature vector and the language unit vector, This includes inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target speech corresponding to the speech to be processed or the text feature vector.

[0010] In an exemplary embodiment of the present invention, inputting the speech feature vector into a speech generation model to obtain a language unit vector is, The speech feature vector is input to the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech. The process includes performing a loss calculation on the audio to be processed and the self-reduced audio to obtain a first loss value, and determining the speech coding vector as a language unit vector based on the first loss value.

[0011] In an exemplary embodiment of the present invention, inputting the speech feature vector to the speech generation model such that the speech generation model outputs a speech coding vector and a self-reduced speech is: The aforementioned speech feature vector is input to a speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector. The speech encoding vector is quantized using the vector quantization module of the speech generation model to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained. The method includes obtaining self-reduced speech by nonlinearly transforming the speech quantization sequence and the speaker vector using the decoder module of the speech generation model.

[0012] In an exemplary embodiment of the present invention, obtaining the speaker vector corresponding to the audio to be processed is: Obtaining a speaker identifier corresponding to the target voice to be processed and determining a correspondence relationship between the speaker identifier and a speaker vector, wherein the correspondence relationship is determined based on the voice generation model, including querying the speaker vector corresponding to the speaker identifier based on the correspondence relationship.

[0013] In an exemplary embodiment of the present invention, obtaining the voice quantization sequence by quantizing the voice encoding vector by the vector quantization module of the voice generation model includes quantizing the voice encoding vector by a nearest neighbor search algorithm based on a codebook in the vector quantization module of the voice generation model to obtain a voice quantization sequence.

[0014] In an exemplary embodiment of the present invention, quantizing the voice encoding vector by the nearest neighbor search algorithm to obtain a voice quantization sequence includes obtaining an updated codebook by updating the codebook, and quantizing the voice encoding vector by a nearest neighbor search algorithm based on the updated codebook to obtain a voice quantization sequence.

[0015] In an exemplary embodiment of the present invention, obtaining an updated codebook by updating the codebook includes obtaining a codebook identifier of the codebook for each frame, comparing the codebook identifiers to obtain a comparison result, and merging the codebooks according to the comparison result to obtain an updated codebook.

[0016] In an exemplary embodiment of the present invention, inputting the target feature vector to a sequence-to-sequence model to obtain an acoustic feature vector Obtain the acoustic vector to be processed of the feature vector to be processed, and by inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, the sequence-to-sequence model outputs a processed acoustic vector, and Perform loss calculation on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and determine the processed acoustic vector as an acoustic feature vector based on the second loss value.

[0017] In an exemplary embodiment of the present invention, the sequence-to-sequence model outputs a processed acoustic vector by inputting the feature vector to be processed and the acoustic vector to be processed into the sequence-to-sequence model, which is Input the feature vector to be processed and the acoustic vector to be processed into the sequence-to-sequence model, and non-linearly map the feature vector to be processed and the acoustic vector to be processed by the encoder module of the sequence-to-sequence model to obtain a spatial encoding vector, and Add the spatial encoding vector and the speaker vector to obtain an alignment target vector, and obtain an audio feature sequence, and Align the alignment target vector and the audio feature sequence by the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and non-linearly map the context representation vector by the decoder of the sequence-to-sequence model to obtain a processed acoustic vector.

[0018] In an exemplary embodiment of the present invention, determining the feature vector to be processed based on the text feature vector and the language unit vector is Determine the text feature vector or the language unit vector as the feature vector to be processed, or This includes adding the text feature vector and the language unit vector to obtain the feature vector to be processed.

[0019] In an exemplary embodiment of the present invention, inputting the acoustic feature vector to a vocoder to obtain the target speech corresponding to the processed speech or the text feature vector is: The process involves extracting the audio acoustic features of the acoustic feature vector via a post-processing network, inputting the audio acoustic features into a vocoder, and having the vocoder output pending audio corresponding to the audio to be processed or the text feature vector. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain a third loss value, and determining the pending audio as the target audio based on the third loss value.

[0020] In an exemplary embodiment of the present invention, performing loss calculations on the pending audio and the audio to be processed to obtain a third loss value is: If the vocoder is a generative adversarial network, loss calculations are performed on the pending audio and the audio to be processed to obtain the generative adversarial network loss value of the generative adversarial network. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain an audio feature loss value, and obtaining a third loss value by weighting and adding the adversarial network loss value and the audio feature loss value.

[0021] According to a second embodiment of the present invention, a voice generation device is provided, and the voice generation device is A data acquisition module configured to acquire speech feature vectors of the speech to be processed, input the speech feature vectors into a speech generation model to acquire language unit vectors, A vector determination module configured to acquire a text feature vector and determine a feature vector to be processed based on the text feature vector and the language unit vector, The system includes a speech generation module configured to input the processing target feature vector into a sequence-to-sequence model to obtain an acoustic feature vector, and to input the acoustic feature vector into a vocoder to obtain the processing target speech or a target speech corresponding to the text feature vector.

[0022] According to a third embodiment of the present invention, an electronic device is provided that includes a processor and a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, a speech generation method in any of the exemplary embodiments described above is realized.

[0023] According to a fourth embodiment of the present invention, a computer-readable storage medium is provided in which a computer program is stored, and when executed by a processor, the computer program realizes the speech generation method in any of the exemplary embodiments described above.

[0024] As can be seen from the above technical proposals, the speech generation method, speech generation apparatus, computer storage medium, and electronic device in the exemplary embodiments of this disclosure offer at least the following advantages and excellent effects.

[0025] In the exemplary embodiments of this disclosure, the method and apparatus can receive speech and text as input by acquiring speech feature vectors and text feature vectors, integrating a speech synthesis task and a speech timbre conversion task, performing multimodal modeling, and improving the performance of the speech synthesis task and the speech timbre conversion task. Furthermore, for small amounts of data, the method and apparatus can acquire speech feature vectors and text feature vectors, providing multiple timbre cloning strategies, improving the effectiveness of timbre cloning with small amounts of data, reducing the difficulty and time of training various models, and supporting timbre cloning methods in various application scenarios.

[0026] The above general statements and subsequent detailed explanations are illustrative and explanatory only and do not limit this disclosure. [Brief explanation of the drawing]

[0027] [Figure 1] This is a schematic flowchart illustrating the speech generation method in an exemplary embodiment of the present disclosure. [Figure 2] This flowchart schematically illustrates how a speech generation model in an exemplary embodiment of the present disclosure outputs language unit vectors. [Figure 3] This flowchart schematically illustrates the processing method of the speech generation model in an exemplary embodiment of the present disclosure. [Figure 4] This flowchart schematically illustrates a method for quantizing an audio quantization sequence in an exemplary embodiment of the present disclosure. [Figure 5] This flowchart schematically illustrates a method for updating a codebook in an exemplary embodiment of the present disclosure. [Figure 6] This flowchart schematically illustrates the method for obtaining speaker vectors in the exemplary embodiments of this disclosure. [Figure 7] This flowchart schematically illustrates a method for determining the feature vector to be processed based on the text feature vector and language unit vector in an exemplary embodiment of the present disclosure. [Figure 8] This flowchart schematically illustrates a method for outputting acoustic feature vectors from a sequence-to-sequence model in an exemplary embodiment of the present disclosure. [Figure 9] This flowchart schematically illustrates the processing method of a sequence-to-sequence model in an exemplary embodiment of the present disclosure. [Figure 10] This flowchart schematically illustrates the method by which the generator outputs the target sound in an exemplary embodiment of the present disclosure. [Figure 11] This flowchart schematically illustrates a method for obtaining a third loss value by performing loss calculations in an exemplary embodiment of the present disclosure. [Figure 12]This figure schematically illustrates the framework of a speech generation model in an application scenario in an exemplary embodiment of the present disclosure. [Figure 13] This is a schematic diagram of the speech generation device in an exemplary embodiment of the present disclosure. [Figure 14] A schematic diagram shows an electronic device for realizing the speech generation method in the exemplary embodiment of this disclosure. [Figure 15] A computer-readable storage medium for realizing the speech generation method in the exemplary embodiments of this disclosure is schematically shown. [Modes for carrying out the invention]

[0028] Hereinafter, these embodiments will be described in more detail with reference to the drawings. However, the exemplary embodiments can be implemented in various forms and are not limited to the embodiments described herein. These embodiments are provided to make the disclosure more comprehensive and complete and to comprehensively convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable way into one or more embodiments. Many specific details are provided in the following description to fully understand the embodiments of the disclosure. However, those skilled in the art will recognize that in practicing the technical ideas of the disclosure, one or more of the aforementioned specific details may be omitted, or other methods, components, apparatus, steps, etc., may be adopted. In other cases, prior art ideas are not described or shown in detail to avoid ambiguity in any aspect of the disclosure.

[0029] In this specification, the terms “one,” “1,” “the,” and “the foregoing” are used to indicate the presence of one or more elements / components / etc., the terms “including” and “having” mean open inclusion and may include other elements / components / etc. in addition to those listed, and the terms “first,” “second,” etc. are used solely as markers and do not limit the number of objects.

[0030] The drawings are schematic representations of this disclosure and are not necessarily drawn proportionally. Identical or corresponding parts in the drawings are denoted by the same reference numerals, and redundant explanations are omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0031] In recent years, with the rapid development of deep learning, speech synthesis technology has made remarkable progress. Simultaneously, the development of various deep learning technologies has also led to rapid advancements in speech timbre conversion. However, both TTS and VC models require a large amount of recorded speech data (more than 10 hours) to achieve the desired results. However, recording speech is extremely expensive and cumbersome. Therefore, guaranteeing the effectiveness of speech synthesis and timbre conversion even when only a small amount of speaker data is available has become a hot topic in research. This research is called speaker adaptation or speaker cloning.

[0032] Here, speaker adaptation is a technique that enables a deep learning model to rapidly and automatically adapt to a target speaker, thereby significantly improving the performance of that speaker.

[0033] From the perspective of speaker cloning, speech synthesis technology and speech timbre conversion should be considered systems that generate the voice of a target speaker based on different inputs.

[0034] In the field of TTS (Text-to-Speaker Simulation), speaker adaptation can be divided into supervised speaker adaptation and unsupervised speaker adaptation.

[0035] Here, supervised speaker adaptation refers to data that requires a <text, audio> pair for adaptation, while unsupervised speaker adaptation refers to data that requires only audio data for adaptation and does not require corresponding text.

[0036] As previous research has shown, supervised speaker adaptation can achieve high-quality results by fine-tuning a basic multi-speaker model using a small amount of target speaker text-to-audio data.

[0037] However, with unsupervised speaker adaptation, it is not possible to fine-tune the model.

[0038] A common unsupervised speaker adaptation method involves using a speaker identification model to extract a single speaker vector from a speech segment, and then using that speaker vector to synthesize the speaker's speech.

[0039] However, as the amount of data increases, the performance of this method does not improve any further.

[0040] Currently, some research is being conducted on combining speech synthesis technology with speech timbre conversion.

[0041] For example, one could use a sequence-to-sequence TTS model to extract speaker-independent expressions and model a VC model, then use a TTS pre-training model to improve the VC model's effectiveness, encode different input source content using different encoders, and then decode it using the same decoder, thus handling two tasks simultaneously. However, in terms of effectiveness, the TTS's effectiveness is always diminished.

[0042] Alternatively, different encoders can be used to encode different input source content, but this requires complex model training and numerous loss functions and hyperparameters. Consequently, research combining speech synthesis and speech timbre conversion is not only complex in its training methods, but it also fails to handle speech synthesis and speech timbre conversion effectively, making it impossible to simultaneously improve the performance of both tasks.

[0043] In response to problems present in related technologies, this disclosure provides a speech generation method. Figure 1 shows a flowchart of the speech generation method, and as shown in Figure 1, the speech generation method includes at least the following steps.

[0044] In step S110, the speech feature vector of the speech to be processed is obtained, and the speech feature vector is input into the speech generation model to obtain the language unit vector.

[0045] In step S120, a text feature vector is obtained, and the feature vector to be processed is determined based on the text feature vector and the language unit vector.

[0046] In step S130, the feature vector to be processed is input to a sequence-to-sequence model to obtain an acoustic feature vector, and the acoustic feature vector is input to a vocoder to obtain a target speech corresponding to the speech or text feature vector to be processed.

[0047] In the exemplary embodiments of this disclosure, speech and text can be received as input by acquiring speech and text feature vectors, merging speech synthesis and speech timbre conversion tasks, performing multimodal modeling, and improving the performance of both tasks. Furthermore, for small amounts of data, speech and text feature vectors are acquired, providing multiple timbre cloning strategies, improving the effectiveness of timbre cloning with small amounts of data, reducing the difficulty and time required to train various models, and supporting timbre cloning methods in various application scenarios.

[0048] The following describes each step of the speech generation method in detail.

[0049] In step S110, the speech feature vector of the speech to be processed is obtained, and the speech feature vector is input into the speech generation model to obtain the language unit vector.

[0050] In exemplary embodiments of this disclosure, the audio to be processed may be audio to be converted in order to perform audio timbre conversion.

[0051] Here, the voice timbre conversion system automatically converts speaker A's voice to speaker B's voice while preserving the content of the speech.

[0052] Therefore, the audio being processed can be understood as the audio of speaker A.

[0053] Accordingly, the speech feature vector of the audio to be processed may be a Mel frequency spectrum feature vector extracted based on the audio to be processed.

[0054] Mel frequency spectrum features can better embody human auditory characteristics by simulating, to some extent, the processing characteristics of the human ear for sound, thereby improving the user's auditory experience.

[0055] After obtaining the speech feature vector, the speech feature vector can be input into the speech generation model so that the speech generation model outputs the corresponding language unit vector.

[0056] In one of the selectable embodiments, Figure 2 is a flowchart of a method by which a speech generation model outputs a language unit vector, and as shown in Figure 2, the method includes at least the following steps, in step S210, a speech feature vector is input to the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech.

[0057] In one of the selectable embodiments, Figure 3 is a flowchart of a method for processing a speech generation model, and as shown in Figure 3, the method includes at least the following steps: in step S310, a speech feature vector is input to the speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector.

[0058] Here, the speech generation model may be a VQ-VAE (Vector Quatization-Variational AutoEncoder) model or another model, and this exemplary embodiment is not particularly limited to it.

[0059] The VQ-VAE model is an autoencoder characterized by the discrete nature of its encoded vectors. A VQ-VAE includes an encoding layer and a decoding layer. The encoding layer encodes speech feature vectors into discrete encoded vectors, and the decoding layer decodes these discrete encoded vectors back into vectors.

[0060] Specifically, if the speech generation model is a VQ-VAE model, the speech feature vector is input to the speech generation model, and the encoder module in the VQ-VAE model maps the speech feature vector to a high-dimensional speech coding vector using a nonlinear transformation. This speech coding vector is z 1:N It can be expressed as follows.

[0061] This encoder can extract more abstract, higher-dimensional features from the input features through nonlinear transformations of a neural network.

[0062] Here, the encoder module in the VQ-VAE model may consist of a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory).

[0063] Specifically, a CNN model may include an input layer, a convolutional layer, a pooling layer, a fully connected layer (FC), and an output layer.

[0064] Here, the convolutional layer uses the ReLU (Normalized Linear Activation Function) as its activation function, while the pooling layer has no activation function. The combination of convolutional and pooling layers can appear multiple times in the hidden layer, and in practice, this number is determined according to the model's requirements.

[0065] Of course, you can use combinations of convolutional layers, or combinations of convolutional layers and pooling layers, and there are no restrictions on how you build the model. However, the most common CNN is a combination of several convolutional layers and pooling layers.

[0066] After several convolutional and pooling layers, there is a fully connected layer, which is essentially a DNN (Deep Neural Network) structure, with the output layer simply performing tasks such as classification using the Softmax activation function.

[0067] LSTM is a special type of RNN (Recurrent Neural Network) primarily designed to solve the problems of gradient vanishing and gradient explosion in long sequence training processes. Simply put, compared to general RNNs, LSTMs can provide better representation in longer sequences.

[0068] In step S320, the speech coding vector is quantized by the speech generation model's vector quantization module to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained.

[0069] The encoder module of the VQ-VAE model uses the audio coding vector z 1:N After obtaining the vector quantization in the VQ-VAE model, The TIFF0007879274000001.tif7170 module is used to quantize a high-dimensional speech coding vector into a speech quantization sequence. This speech quantization sequence is: It may also be represented as TIFF0007879274000002.tif7170.

[0070] In selectable embodiments, a speech encoding vector is quantized using a nearest neighbor search algorithm based on the codebook in the vector quantization module of the speech generation model to obtain a speech quantization sequence.

[0071] Specifically, based on the codebook in the VQ-VAE model, a nearest neighbor search algorithm is used to quantize a series of speech coding vectors into a discrete speech quantization sequence.

[0072] In one of the selectable embodiments, Figure 4 is a flowchart of a method for quantizing an audio quantization sequence, and as shown in Figure 4, the method includes at least the following steps, in step S410, updating the codebook to obtain the updated codebook.

[0073] In one of the selectable embodiments, Figure 5 is a flowchart of a method for updating a codebook, which, as shown in Figure 5, includes at least the following steps: in step S510, the codebook identifier of the codebook for each frame is obtained, and the codebook identifiers are compared to obtain the comparison result.

[0074] Since the audio feature vectors for each frame correspond to their respective codebooks, for example, the codebooks for the audio feature vectors of 5 frames may be book1, book1, book1, book2, and book2.

[0075] Considering that text feature vectors may be processed in subsequent, sequence-to-sequence models, we can compare codebook identifiers to obtain a more fitted sequence representation of the text feature vectors and retrieve the comparison results.

[0076] Here, the comparison result can reflect whether the preceding and succeeding two or more codebook identifiers are the same.

[0077] In step S520, the codebooks are merged based on the comparison results to obtain the updated codebook.

[0078] If the comparison results reflect that two or more codebook identifiers are the same, the same codebook identifiers can be merged into one, as queryed by the nearest neighbor search algorithm.

[0079] For example, if the codebooks for 5 frames of audio feature vectors are book1, book1, book1, book2, and book2, the codebook identifiers can be merged into book1 and book2 to create an updated codebook.

[0080] In this exemplary embodiment, updating the codebook via a post-processing network allows the speech quantization sequence to obtain a sequence representation that better fits the text feature vector, providing data support for multimodal speech-to-timbre conversion tasks and being useful for improving speech-to-timbre conversion performance.

[0081] In step S420, based on the updated codebook, the speech coding vector is quantized using a nearest neighbor search algorithm to obtain the speech quantization sequence.

[0082] Here, each updated codebook is a K*D dimension codebook maintained in the VQ-VAE model.

[0083] For example, each codebook contains K D-dimensional coding vectors e1, e2, ..., e KIt may include: Encoding an H'*W'*D dimension speech feature vector using the encoding layer of the VQ-VAE model, and further encoding the closest D-dimensional vector e in the H'*W'*D dimension speech feature vector. i These can be found in the codebook. Here, the coded vector e i This is a vector in the codebook, and the D-dimensional vector is encoded by vector e i By representing it using an index, we obtain a discrete vector of dimension H'*W'. Here, K, D, H', and W' represent the dimensions, respectively.

[0084] Furthermore, based on a pre-configured discrete coding scheme, the H'*W' dimensional discrete vector is converted into an audio quantization sequence.

[0085] Here, the pre-configured discrete coding scheme may be one-hot coding or other types of coding schemes, and this exemplary embodiment is not particularly limited to these.

[0086] Specifically, using a one-hot coding codebook, an H'*W' dimensional discrete vector is transformed into another H'*W' dimensional discrete coded vector encoded by the one-hot coding codebook using a lookup table method, and then a speech quantization sequence is obtained based on the transformed H'*W' dimensional discrete coded vector.

[0087] For example, one 3x3 discrete vector can be converted into a 3x3 discrete coded vector encoded with a different one-hot coding scheme codebook, and then a 1x9 speech quantization sequence can be obtained based on each element in the converted 3x3 discrete coded vector.

[0088] In this exemplary embodiment, the speech generation model's vector quantization module performs discrete coding on the speech coding vector to obtain the corresponding speech quantization sequence, and provides a database and theoretical support to enable the speech generation model to output the speech coding vector and the self-reduced speech.

[0089] In addition, the speaker vector of the speaker who uttered or pronounced the audio to be processed may be obtained.

[0090] In one of the selectable embodiments, Figure 6 is a flowchart of a method for obtaining speaker vectors, which, as shown in Figure 6, includes at least the following steps: in step S610, a speaker identifier corresponding to the speech to be processed is obtained, and a correspondence between the speaker identifier and the speaker vector is determined, where the correspondence is determined based on a speech generation model.

[0091] Here, the speaker identifier can uniquely represent the identifier information of the speaker who uttered or pronounced the audio to be processed.

[0092] Furthermore, the first loss value calculated between the self-reduced speech output by the decoder in the speech generation model and the speech to be processed allows for the simultaneous maintenance of a table that stores the correspondence between speaker identifiers and speaker vectors.

[0093] In step S620, the speaker vector corresponding to the speaker identifier is queried based on the correspondence.

[0094] In a table that stores the correspondence between speaker identifiers and speaker vectors, it is possible to query the corresponding speaker vector based on the speaker identifier.

[0095] In this exemplary embodiment, data support is provided to the decoder module of the speech generation model by obtaining a speaker vector maintained by the speech generation model, and the support provided by training the decoder module and encoder module via the speaker vector and speech quantization sequence can be useful for generating the encoder and determining the speaker vector of the speech generation model.

[0096] In step S330, the decoder module of the speech generation model performs a nonlinear transformation on the speech quantization sequence and speaker vector to obtain self-reduced speech.

[0097] The decoder module of the speech generation model can receive a quantized speech quantization sequence, obtain a speaker vector, add the speech quantization sequence and the speaker vector, and then reduce it through a nonlinear change to obtain self-reduced speech.

[0098] This decoder transforms high-dimensional, abstract hidden features into more explicit features through nonlinear transformations of neural networks.

[0099] Furthermore, if the speech generation model is a VQ-VAE model, the decoder module in the VQ-VAE model may consist of a CNN and an LSTM.

[0100] In this exemplary embodiment, the corresponding processing of the encoder module, vector quantization module, and decoder module in the speech generation model can output self-reduced speech to support the training of the speech generation model.

[0101] In step S220, loss calculations are performed on the audio to be processed and the self-reduced audio to obtain a first loss value, and the speech coding vector is determined as the language unit vector based on the first loss value.

[0102] After the voice generation model outputs the self-return voice, the first loss value of the voice generation model can be calculated for the voice to be processed and the self-return voice.

[0103] Specifically, the first loss value may be calculated by an L2 norm loss function.

[0104] The L2 norm loss function is shown in Equation (1). TIFF0007879274000003.tif11170

[0105] The L2 norm loss function is also called the least square error (LSE). The L2 norm loss function minimizes the sum of the squares of the differences between the target value y i and the estimated value f(x i ).

[0106] In a general regression problem, this loss is used, and outliers have a large impact on this loss.

[0107] When the first loss value calculated according to Equation (1) reaches a stable value and does not decrease any further, it indicates that the training of the voice generation model has already been completed.

[0108] In this case, the voice encoding vector output from the encoder module in the voice generation model can be determined as the teacherless language unit vector. The language unit vector can be represented by k i .

[0109] In this exemplary embodiment, the voice generation model can obtain the language unit vector, provide the data input of the voice modality to the sequence-to-sequence model, and support the multi-module voice generation method.

[0110] In step S120, a text feature vector is obtained, and a feature vector to be processed is determined based on the text feature vector and the language unit vector.

[0111] In exemplary embodiments of this disclosure, speech synthesis is a system that automatically converts natural text into speech.

[0112] Therefore, after obtaining natural text, the phonemesequences of the natural text can be extracted as text feature vectors. The method for extracting the phonemesequences of the natural text may be implemented using an LSTM model, and this exemplary embodiment is not particularly limited to that.

[0113] After obtaining the text feature vector, the feature vector to be processed can be determined based on the text feature vector and the language unit vector.

[0114] In an optional embodiment, Figure 7 shows a flowchart of a method for determining the feature vector to be processed based on a text feature vector and a language unit vector, and as shown in Figure 7, the method includes at least the following steps, in step S710, the text feature vector or language The unit vector is determined as the feature vector to be processed.

[0115] In the field of TTS (Text-to-Speech), speaker adaptation can be divided into supervised speaker adaptation and unsupervised speaker adaptation. Here, supervised speaker adaptation refers to data that requires a <text, audio> pair for adaptation, while unsupervised speaker adaptation refers to data that requires only audio data and not corresponding text for adaptation.

[0116] As previous research has shown, supervised speaker adaptation can achieve high-quality results by fine-tuning a basic multi-speaker model using a small amount of target speaker text-to-audio data.

[0117] However, with unsupervised speaker adaptation, it is not possible to fine-tune the model.

[0118] Here, a typical unsupervised speaker adaptation method extracts a single speaker vector from a speech segment using a speaker identification model, and then synthesizes the speaker's speech using that speaker vector.

[0119] Clearly, unsupervised speaker adaptation cannot be achieved by trained models. Furthermore, the performance of this method does not improve further as the amount of data increases. Therefore, in unsupervised speaker adaptation, the text feature vector can be determined as the feature vector to be processed, and the text synthesis effect can be realized by the subsequent sequence-to-sequence model and vocoder.

[0120] Voice tone conversion is, language The unit vector is determined as the feature vector to be processed, and the subsequent sequence-to-sequence model and vocoder can be used to achieve the speech timbre conversion task.

[0121] In step S720, the text feature vector and the language unit vector are added together to obtain the feature vector to be processed.

[0122] To improve the effectiveness of the speech timbre conversion task, the text feature vector and the language unit vector can be added together to obtain the feature vector to be processed.

[0123] Furthermore, since the codebook in the speech generation model's vector quantization module has already been updated during the process of obtaining the language unit vectors, the language unit vectors and the text feature vectors are a very good match, and therefore the text feature vectors and language unit vectors can be directly added together.

[0124] When a processed feature vector is obtained by adding a text feature vector and a language unit vector, this processed feature vector corresponds to a representation that adds text modality to language modality; therefore, this processed feature vector is an enhanced data representation. Based on this, the performance of the speech timbre conversion task realized by this processed feature vector is better.

[0125] Here, modality refers to the source or format of information.

[0126] For example, information may be expressed in various forms such as audio, video, text, and images, and each of these forms of expression may be called a type of modality of that information. Based on this, multimodal information is fused in ways such as text, audio, visual, action, and environment. Multimodal machine learning (MMML) is the ability to process and understand multi-source modality information using machine learning methods, and for example, a currently popular research direction is multimodal learning between images, videos, audio, and semantics.

[0127] In this exemplary embodiment, different feature vectors to be processed can be determined as the basis for subsequent model processing based on the difference between the speech synthesis task and the speech timbre conversion task, providing data support to improve the performance of the speech synthesis task and the speech timbre conversion task.

[0128] In step S130, the feature vector to be processed is input to a sequence-to-sequence model to obtain an acoustic feature vector, and the acoustic feature vector is input to a vocoder to obtain a target speech corresponding to the speech or text feature vector to be processed.

[0129] In the exemplary embodiments of this disclosure, after determining the feature vector to be processed, the corresponding acoustic feature vector can be obtained by inputting the feature vector to be processed into a sequence-to-sequence model.

[0130] In one of the selectable embodiments, Figure 8 shows a flowchart of a method by which a sequence-to-sequence model outputs an acoustic feature vector, and as shown in Figure 8, the method includes at least the following steps: in step S810, the acoustic vector to be processed is obtained from the feature vector to be processed, and the feature vector to be processed and the acoustic vector to be processed are input to the sequence-to-sequence model, so that the sequence-to-sequence model outputs a processed acoustic vector.

[0131] The acoustic vector to be processed may also be a Mel-frequency spectrum feature vector.

[0132] In one of the selectable embodiments, Figure 9 shows a flowchart of a sequence-to-sequence model processing method, which, as shown in Figure 9, includes at least the following steps: in step S910, the feature vector to be processed and the acoustic vector to be processed are input to the sequence-to-sequence model, and the encoder module of the sequence-to-sequence model performs a nonlinear mapping on the feature vector to be processed and the acoustic vector to be processed to obtain a spatially encoded vector.

[0133] The sequence-to-sequence model may be a sequence-to-sequence (Seq2seq) model of an attention mechanism, or any other model, and this exemplary embodiment is not particularly limited thereto.

[0134] If the sequence-to-sequence model is a Seq2seq model based on an attention mechanism, the sequence-to-sequence model may include an encoder module, an attention mechanism, and a decoder module.

[0135] Here, the encoder module of the sequence-to-sequence model may be configured to acquire a representation sequence corresponding to the feature vector to be processed and the acoustic vector to be processed, the attention mechanism may be configured to generate a fixed-length semantic representation according to the representation sequence, and the decoder module may acquire the acoustic vector according to the semantic representation.

[0136] Specifically, the encoder module of a sequence-to-sequence model may include a feature embedding layer, a convolutional pre-net, a dense pre-net, a CBHG (Convolution Bank + Highway network + bidirectional Gated Recurrent Unit, i.e., a convolutional layer + highway network + bidirectional recurrent neural network, i.e., CBHG consists of a convolutional layer, a highway network and a bidirectional recurrent neural network) submodel, and a down-sampling convolution layer.

[0137] First, the feature vector to be processed is encoded using a FeatureEmbedding layer and then input into a Convolutional Pre-net. By performing a nonlinear transformation on the encoded feature vector and the acoustic vector to be processed, the convergence and generalization capabilities of the sequence-to-sequence model based on the attention mechanism are improved. Simultaneously, the number of audio frames corresponding to the acoustic vector to be processed is input into a Dense Pre-net to obtain the corresponding depth features. Then, the outputs of the Convolutional Pre-net and the Dense Pre-net are input together into a CBHG submodel to extract the corresponding context features. These are then input into a Down-sampling Convolution to reduce the computational complexity and receptive field, ultimately obtaining the appropriate spatially encoded vector.

[0138] Therefore, a sequence-to-sequence model encoder module performs a nonlinear transformation on the feature vector and acoustic vector to be processed and maps them to a high-dimensional spatial coding vector. This spatial coding vector is h t It can be expressed as follows.

[0139] In step S920, the spatial coding vector and the speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is acquired.

[0140] To perform multi-speaker modeling, the attention mechanism of the sequence-to-sequence model can also receive speaker vectors as input.

[0141] To input the speaker vector, the spatially encoded vector and the speaker vector can be added together to obtain the alignment target vector.

[0142] Furthermore, since the attention mechanism is an autoregressive model, it is also possible to obtain a speech feature sequence. This speech feature sequence is m t It can be represented as -1. When t=1, the speech feature sequence is initialized to a sequence of all zeros, and at t=2 and thereafter, the speech feature sequence is a feedback sequence for the decoder module to the previous time.

[0143] In step S930, the alignment target vector and the audio feature sequence are aligned by the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and a nonlinear mapping is performed on the context representation vector by the decoder of the sequence-to-sequence model to obtain a processed acoustic vector.

[0144] Typically, the speech feature vector is longer than the alignment target vector, so the context representation vector can be obtained by aligning the alignment target vector and the speech feature sequence.

[0145] Specifically, the method for aligning the alignment target vector and the speech feature sequence may involve performing a dot product calculation on the alignment target vector and the speech feature sequence.

[0146] Furthermore, the contextual representation vector obtained by aligning the alignment target vector and the speech feature sequence reflects the contextual relationships of the context, ensuring the effectiveness of speech generation.

[0147] Furthermore, the sequence-to-sequence model decoder module primarily obtains a processed acoustic vector by returning the context representation vector, which is obtained by performing alignment according to the alignment target vector and the speech feature sequence, back to the original speech acoustic feature space through nonlinear mapping. Therefore, the processed acoustic vector may be a Mel frequency spectrum, and the processed acoustic vector may be represented by m.

[0148] In this exemplary embodiment, the correspondence processing of the encoder module, attention mechanism, and decoder module in the sequence-to-sequence model between the feature vector to be processed and the acoustic vector to be processed provides a fusion method for speech synthesis and speech timbre conversion tasks, improving the timbre cloning effect with small amounts of data. Furthermore, since multiple types of input data can be received, timbre cloning in a variety of scenes is supported.

[0149] In step S820, a loss calculation is performed on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and the processed acoustic vector is determined as the acoustic feature vector based on the second loss value.

[0150] After the sequence-to-sequence model outputs the processed acoustic vector, a second loss value can be calculated between the target acoustic vector and the processed acoustic vector according to equation (1).

[0151] If the second loss value calculated according to equation (1) reaches a stable value and does not decrease further, it indicates that training of the sequence-to-sequence model is already complete.

[0152] In this case, the processed acoustic vectors output to the sequence-to-sequence model converged through training can be determined as acoustic feature vectors.

[0153] After the sequence-to-sequence model outputs an acoustic feature vector, this acoustic feature vector can then be input into a vocoder to obtain the target speech for a text synthesis task or speech timbre conversion.

[0154] In one of the selectable embodiments, Figure 10 is a flowchart of a method for outputting target speech by a generator, which, as shown in Figure 10, includes at least the following steps: in step S1010, speech acoustic features of an acoustic feature vector are extracted via a post-processing network, and the speech acoustic features are input to a vocoder, which then outputs pending speech corresponding to the speech or text feature vector to be processed.

[0155] Of these, the post-processing network is primarily configured to generate higher-precision audio characteristics. These audio characteristics are It may also be represented as TIFF0007879274000004.tif7170.

[0156] The post-processing network may be a CNN network, an LSTM network, or the like, and this exemplary embodiment is not particularly limited to these.

[0157] A vocoder is a system that converts acoustic features, such as the Mel frequency spectrum, into speech audio.

[0158] Here, the vocoder may be a Wavenet model, a Griffin-Lim algorithm, a GAN network (Generative Adversarial Network), or the like, and this exemplary implementation is not particularly limited to these.

[0159] Specifically, the Wavenet model is a sequence generation model that can be used for speech generation modeling. In modeling acoustic models for speech synthesis, Wavenet has good synthesis effects because it can directly learn the mapping to sample value sequences. Currently, Wavenet has great potential in the field of speech synthesis for modeling acoustic models for speech synthesis.

[0160] The Wavenet model can predict the result of the t-th point according to the t-1 point prior to a sequence, and therefore can be used to predict the numerical value of sampling points in audio.

[0161] Griffin-Lim is an algorithm that reconstructs speech under the condition that only the amplitude spectrum is known and the phase spectrum is unknown.

[0162] The implementation of the Griffin-Lim algorithm is straightforward. The Griffin-Lim algorithm is an iterative algorithm in which the iteration process first randomly initializes a phase spectrum, then synthesizes a new speech using the phase spectrum and known amplitude spectrum with the ISTFT (Inverse Short-Time Fourier Transform), then performs the STFT (Short-Time Fourier Transform) on the synthesized speech to obtain new amplitude and phase spectra, and finally discards the new amplitude spectrum and synthesizes speech using the phase spectrum and known amplitude spectrum, and so on.

[0163] GAN networks are a machine learning method proposed by Ian J. Goodfello et al. in their 2014 paper, Generative Adversarial Networks. In GAN networks, there are two models: a generative model (G) and a discriminative model (D).

[0164] Taking image generation as an example, G is an image-generating network that receives random noise z, then generates an image using this noise, and the generated data is denoted as G(z).

[0165] D is a discriminant network that identifies whether an image is "true" or not (i.e., whether it is a fake or not). Its input parameter is x, where x represents an image, and its output D(x) represents the probability that x is a true image. An output of 1 indicates that the image is true, while an output of 0 indicates that there is no possibility that the image is true.

[0166] During training, the goal of the generative network G is to deceive the discriminative network D by generating false images, and the goal of the discriminative network D is to determine whether a given image was generated by G or not. This constitutes one game. Simultaneously, the capabilities of both G and D gradually improve during the training process. In the most ideal case, D(G(z)) = 0.5.

[0167] A vocoder can process the audio acoustic features extracted by a post-processing network and output pending audio.

[0168] When performing a speech synthesis task, the pending speech may be speech synthesized based on text feature vectors, and when performing a speech timbre conversion task, the pending speech may be speech converted based on the speech to be processed.

[0169] In step S1020, a loss calculation is performed on the pending audio and the audio to be processed to obtain a third loss value, and the pending audio is determined to be the target audio based on the third loss value.

[0170] In one of the selectable embodiments, Figure 11 shows a flowchart of a method for performing loss calculations to obtain a third loss value, which, as shown in Figure 11, includes at least the following steps: in step S1110, if the vocoder is a generative adversarial network, loss calculations are performed on the pending speech and the speech to be processed to obtain the adversarial network loss value of the generative adversarial network.

[0171] When a vocoder employs a generative adversarial network, the loss function of the generative adversarial network is given by equation (2). TIFF0007879274000005.tif16170

[0172] Here, D(x) identifies the true sample, and since a classification result closer to 1 is desirable, the loss function is log(D(x)), z is a random input, and G(z) represents the generated sample. For the generated sample, the closer the classifier's classification result D(G(z)) is to 0, that is, the more desirable the total value is maximized, so the overall representation is as shown in equation (2).

[0173] Therefore, according to equation (2), loss calculations can be performed on the pending audio and the audio to be processed to obtain the adversarial network loss value of the adversarial generative network.

[0174] In step S1120, loss calculations are performed on the pending audio and the audio to be processed to obtain an audio feature loss value, and a third loss value is obtained by weighting and adding the adversarial network loss value and the audio feature loss value.

[0175] Furthermore, loss calculations can be performed on the pending audio and the audio to be processed according to equation (1) to obtain audio feature loss values.

[0176] Furthermore, based on empirical data, weights corresponding to the adversarial network loss value and the speech feature loss value can be set, and a third loss value can be obtained by weighting and adding the adversarial network loss value and the speech feature loss value.

[0177] Furthermore, if the vocoder employs a different network or model, the corresponding loss value can be calculated as the third loss value according to equation (1) only.

[0178] In this exemplary embodiment, corresponding loss value calculation methods are set based on different vocoder content, ensuring greater compatibility, guaranteeing the accuracy of training results for different types of vocoders, and further guaranteeing the reliability of target speech generation.

[0179] If the third loss value reaches a stable value and does not decrease further, it indicates that the vocoder has already been trained to converge, and since the vocoder is ready for application, the pending speech can be determined to be the target speech.

[0180] The speech generation method in the embodiments of this disclosure will be described in detail below with reference to application scenarios.

[0181] Figure 12 shows a schematic diagram of the speech generation model framework in an application scenario. As shown in Figure 12, the VQ-VAE model includes an encoder module 1210, a vector quantization module 1220, and a decoder module 1230.

[0182] First, the speech feature vectors of the audio are input to the encoder module 1210 of the VQ-VAE model.

[0183] Here, the audio to be processed may be the audio to be converted in order to perform audio timbre conversion. Correspondingly, the audio feature vector of the audio to be processed may be a Mel-frequency spectrum feature vector extracted based on the audio to be processed.

[0184] Furthermore, the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a high-dimensional speech coding vector. This speech coding vector is z 1:N It can be expressed as follows.

[0185] Here, the encoder module in the VQ-VAE model may consist of a CNN and an LSTM.

[0186] Then, the speech generation model's vector quantization module 1220 quantizes the speech coding vector to obtain the speech quantization sequence.

[0187] The encoder module of the VQ-VAE model uses the speech encoding vector z 1:N After obtaining the vector quantization in the VQ-VAE model, The TIFF0007879274000006.tif9170 module quantizes a high-dimensional speech coding vector into a speech quantization sequence. This speech quantization sequence is: It may also be represented as TIFF0007879274000007.tif8170.

[0188] Based on the codebook for the vector quantization module of the speech generation model, the speech coding vector is quantized using a nearest neighbor search algorithm to obtain the speech quantization sequence.

[0189] Specifically, based on the updated codebook in the VQ-VAE model, a nearest neighbor search algorithm is used to quantize a series of speech coding vectors into a discrete speech quantization sequence.

[0190] When updating a codebook, you can obtain the codebook identifier for each frame's codebook, compare the codebook identifiers, and retrieve the comparison result.

[0191] Since the audio feature vectors for each frame correspond to their respective codebooks, for example, the codebooks for the audio feature vectors of 5 frames may be book1, book1, book1, book2, and book2.

[0192] Considering that text feature vectors may be processed in subsequent, sequence-to-sequence models, we can compare codebook identifiers to obtain a more fitted sequence representation of the text feature vectors and retrieve the comparison results.

[0193] Here, the comparison result can reflect whether the preceding and succeeding two or more codebook identifiers are the same.

[0194] Based on the comparison results, the codebooks are merged to obtain the updated codebook.

[0195] If the comparison results reflect that two or more codebook identifiers are the same, the same codebook identifiers can be merged into one, as queryed by the nearest neighbor search algorithm.

[0196] For example, if the codebooks for 5 frames of audio feature vectors are book1, book1, book1, book2, and book2, the codebook identifiers can be merged into book1 and book2 to create an updated codebook.

[0197] Based on the updated codebook, the nearest neighbor search algorithm is used to quantize the speech coding vector and obtain the speech quantization sequence.

[0198] Here, each updated codebook is a K*D dimension codebook maintained in the VQ-VAE model.

[0199] For example, each codebook contains K D-dimensional coding vectors e1, e2, ..., e KIt may include: Encoding an H'*W'*D dimension speech feature vector using the encoding layer of the VQ-VAE model, and further encoding the closest D-dimensional vector e in the H'*W'*D dimension speech feature vector. i These can be found in the codebook. Here, the coded vector e i This is a vector in the codebook, and the D-dimensional vector is encoded by vector e i By representing it using an index, we obtain a discrete vector of dimension H'*W'. Here, K, D, H', and W' represent the dimensions, respectively.

[0200] Furthermore, based on a pre-configured discrete coding scheme, the H'*W' dimensional discrete vector is converted into an audio quantization sequence.

[0201] Here, the pre-configured discrete coding scheme may be one-hot coding or other types of coding schemes, and this exemplary embodiment is not particularly limited to these.

[0202] Specifically, using a one-hot coding codebook, an H'*W' dimensional discrete vector is transformed into another H'*W' dimensional discrete coded vector encoded by the one-hot coding codebook using a lookup table method, and then a speech quantization sequence is obtained based on the transformed H'*W' dimensional discrete coded vector.

[0203] For example, one 3x3 discrete vector can be converted into a 3x3 discrete coded vector encoded with a different one-hot coding scheme codebook, and then a 1x9 speech quantization sequence can be obtained based on each element in the converted 3x3 discrete coded vector.

[0204] In addition, the speaker vector of the speaker who uttered or pronounced the audio to be processed may be obtained.

[0205] The speaker identifier corresponding to the audio to be processed is obtained, and the correspondence between the speaker identifier and the speaker vector is determined, where the correspondence is determined based on the speech generation model.

[0206] Here, the speaker identifier can uniquely represent the identifier information of the speaker who uttered or pronounced the audio to be processed.

[0207] Furthermore, the first loss value calculated between the self-reduced speech output by the decoder in the speech generation model and the speech to be processed allows for the simultaneous maintenance of a table that stores the correspondence between speaker identifiers and speaker vectors.

[0208] The speaker vector corresponding to the speaker identifier is queried based on the correspondence.

[0209] In a table that stores the correspondence between speaker identifiers and speaker vectors, it is possible to query the corresponding speaker vector based on the speaker identifier.

[0210] Finally, the speech generation model's decoder module performs a nonlinear transformation on the speech quantization sequence and speaker vector to obtain self-reduced speech.

[0211] The decoder module of the speech generation model can receive a quantized speech quantization sequence, obtain a speaker vector, add the speech quantization sequence and the speaker vector, and then reduce it through a nonlinear change to obtain self-reduced speech.

[0212] Loss calculations are performed on the audio to be processed and the self-reduced audio to obtain a first loss value, and the speech coding vector is determined as the language unit vector based on the first loss value.

[0213] After the speech generation model outputs a self-reduced speech, the first loss value of the speech generation model can be calculated for both the target speech and the self-reduced speech.

[0214] Specifically, the first loss value may be calculated using the L2-norm loss function. The L2-norm loss function is shown in equation (1).

[0215] If the first loss value calculated according to equation (1) reaches a stable value and does not decrease further, it indicates that the speech generation model has already been trained.

[0216] In this case, the speech coding vector output from the encoder module in the speech generation model can be determined as an unsupervised language unit vector. This language unit vector is k i It can be expressed as follows.

[0217] Unsupervised learning can discover or extract useful information representations from its own data, and in this application scenario, the VQ-VAE unsupervised algorithm can extract discrete information representations from data in different formats.

[0218] Because such discrete representation units and phonemes in language texts are sufficiently similar, it is highly suitable to use these unsupervised, discrete language units as input to end-to-end language synthesis models.

[0219] Furthermore, this also aligns well with the issues that need to be addressed.

[0220] To combine speech synthesis and speech timbre conversion tasks into a single system, we can find common inputs for this system: phonemes extracted from text and unsupervised language units extracted from a VQ-VAE model.

[0221] In Figure 13, the sequence-to-sequence model may include an encoder module 1240, an attention mechanism 1250, a decoder module 1260, and a post-processing network 1270.

[0222] To merge speech synthesis and speech timbre conversion tasks, text feature vectors can also be obtained.

[0223] After obtaining natural text, the phoneme sequence of that natural text can be extracted as a text feature vector.

[0224] After obtaining the text feature vector, the feature vector to be processed can be determined based on the text feature vector and the language unit vector.

[0225] In unsupervised speaker adaptation, text feature vectors are determined as the feature vectors to be processed, and text synthesis effects can be achieved using subsequent sequence-to-sequence models and vocoders.

[0226] Voice tone conversion is, language The unit vector is determined as the feature vector to be processed, and the subsequent sequence-to-sequence model and vocoder can be used to achieve the speech timbre conversion task.

[0227] To improve the effectiveness of the speech timbre conversion task, the text feature vector and the language unit vector can be added together to obtain the feature vector to be processed.

[0228] Furthermore, since the codebook in the speech generation model's vector quantization module has already been updated during the process of obtaining the language unit vectors, the language unit vectors and the text feature vectors are a very good match, and therefore the text feature vectors and language unit vectors can be directly added together.

[0229] When a processed feature vector is obtained by adding a text feature vector and a language unit vector, this processed feature vector corresponds to a representation that adds text modality to language modality; therefore, this processed feature vector is an enhanced data representation. Based on this, the performance of the speech timbre conversion task realized by this processed feature vector is better.

[0230] Furthermore, the acoustic vector to be processed is obtained from the feature vector to be processed, and this acoustic vector to be processed may be a Mel-frequency spectrum feature vector.

[0231] The feature vectors to be processed and the acoustic vectors to be processed are input to the sequence-to-sequence model, and the encoder module 1240 of the sequence-to-sequence model performs a nonlinear mapping on the feature vectors to be processed and the acoustic vectors to be processed to obtain spatially encoded vectors.

[0232] The sequence-to-sequence model may also be a sequence-to-sequence model based on an attention mechanism.

[0233] Specifically, the encoder module of a sequence-to-sequence model may include a feature embedding layer, a convolutional preprocessing network, a dense preprocessing network, a CBHG submodel, and a downsampling convolutional layer.

[0234] First, the feature vector to be processed is encoded using a FeatureEmbedding layer and then input into a Convolutional Pre-net. By performing a nonlinear transformation on the encoded feature vector and the acoustic vector to be processed, the convergence and generalization capabilities of the sequence-to-sequence model based on the attention mechanism are improved. Simultaneously, the number of audio frames corresponding to the acoustic vector to be processed is input into a Dense Pre-net to obtain the corresponding depth features. Then, the outputs of the Convolutional Pre-net and the Dense Pre-net are input together into a CBHG submodel to extract the corresponding context features. These are then input into a Down-sampling Convolution to reduce the computational complexity and receptive field, ultimately obtaining the appropriate spatially encoded vector.

[0235] Therefore, a sequence-to-sequence model encoder module performs a nonlinear transformation on the feature vector and acoustic vector to be processed and maps them to a high-dimensional spatial coding vector. This spatial coding vector is h t It can be expressed as follows.

[0236] The spatial coding vector and speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is then acquired.

[0237] To model a multi-speaker model, the sequence-to-sequence model attention mechanism 1250 can also receive speaker vectors as input.

[0238] To input the speaker vector, the spatially encoded vector and the speaker vector can be added together to obtain the alignment target vector.

[0239] Furthermore, since the attention mechanism 1250 is an autoregressive model, it can also acquire speech feature sequences. These speech feature sequences are m tIt can be represented as -1. When t=1, the audio feature sequence is initialized to a sequence of all zeros, and at t=2 and thereafter, the audio feature sequence is a feedback sequence for the previous time for the decoder module 1260.

[0240] The sequence-to-sequence model's attention mechanism 1250 aligns the alignment target vector with the audio feature sequence to obtain a context representation vector, and the sequence-to-sequence model's decoder 1260 performs a nonlinear mapping on the context representation vector to obtain a processed acoustic vector.

[0241] Typically, the speech feature vector is longer than the alignment target vector, so the context representation vector can be obtained by aligning the alignment target vector and the speech feature sequence.

[0242] Specifically, the method for aligning the alignment target vector and the speech feature sequence may involve performing a dot product calculation on the alignment target vector and the speech feature sequence.

[0243] Furthermore, the contextual representation vector obtained by aligning the alignment target vector and the speech feature sequence reflects the contextual relationships of the context, ensuring the effectiveness of speech generation.

[0244] Furthermore, the sequence-to-sequence model decoder module 1260 primarily obtains a processed acoustic vector by returning the context representation vector, obtained by aligning it according to the alignment target vector and the speech feature sequence, back to the original speech acoustic feature space through nonlinear mapping. Therefore, the processed acoustic vector may be a Mel frequency spectrum, and the processed acoustic vector may be represented by m.

[0245] After the sequence-to-sequence model outputs the processed acoustic vector, a second loss value can be calculated between the target acoustic vector and the processed acoustic vector according to equation (1).

[0246] If the second loss value calculated according to equation (1) reaches a stable value and does not decrease further, it indicates that training of the sequence-to-sequence model is already complete.

[0247] In this case, the processed acoustic vectors output to the sequence-to-sequence model converged through training can be determined as acoustic feature vectors.

[0248] The post-processing network 1270 extracts the speech acoustic features from the acoustic feature vector, and by inputting the speech acoustic features to the vocoder 1280, the vocoder 1280 outputs pending speech corresponding to the speech or text feature vector to be processed.

[0249] Of these, the post-processing network 1270 is primarily configured to generate higher-precision audio characteristics. These audio characteristics are It may also be represented as TIFF0007879274000008.tif7170.

[0250] Among these, the vocoder may be a Wavenet model, a Griffin-Lim algorithm, a GAN network, or the like, and this exemplary embodiment is not particularly limited to these.

[0251] The audio acoustic features extracted by the post-processing network 1270 are processed by the vocoder 1280, which can then output pending audio.

[0252] When performing a speech synthesis task, the pending speech may be speech synthesized based on text feature vectors, and when performing a speech timbre conversion task, the pending speech may be speech converted based on the speech to be processed.

[0253] If the vocoder 1280 is a generative adversarial network, loss calculations are performed on the pending audio and the audio to be processed to obtain the generative adversarial network loss value.

[0254] When the vocoder 1280 employs a generative adversarial network, the loss function of the generative adversarial network is given by equation (2). Therefore, according to equation (2), the loss can be calculated for the pending speech and the speech to be processed to obtain the generative adversarial network loss value.

[0255] Loss calculations are performed on the pending audio and the audio to be processed to obtain audio feature loss values. A third loss value is then obtained by weighting and adding the adversarial network loss value and the audio feature loss value.

[0256] Furthermore, loss calculations can be performed on the pending audio and the audio to be processed according to equation (1) to obtain audio feature loss values.

[0257] Furthermore, based on empirical data, weights corresponding to the adversarial network loss value and the speech feature loss value can be set, and a third loss value can be obtained by weighting and adding the adversarial network loss value and the speech feature loss value.

[0258] Furthermore, if the vocoder 1280 employs a different network or model, the corresponding loss value can be calculated as the third loss value according to equation (1) only.

[0259] If the third loss value reaches a stable value and does not decrease further, it indicates that the vocoder 1280 has already been trained to converge, and since the vocoder 1280 is ready for deployment, the pending voice can be determined to be the target voice.

[0260] According to the voice generation method in the exemplary embodiments of the present disclosure, by obtaining voice feature vectors and text feature vectors, voice and text can be received as inputs, fusing the voice synthesis task and the voice timbre conversion task, performing multimodal modeling, and improving the performance of the voice synthesis task and the voice timbre conversion task. Further, in the case of a small amount of data, voice feature vectors and text feature vectors are obtained, providing strategies for multiple types of timbre clones, improving the effect of timbre clones with a small amount of data, reducing the training difficulty and training time of multiple models, and supporting the timbre cloning method in multiple application scenarios.

[0261] In addition, the voice generation method in this application scenario can obtain good effects through fine-tuning in teacher-guided speaker adaptation, and the performance can also be improved as the amount of data increases in speaker-independent speaker adaptation.

[0262] In teacher-guided timbre cloning, such performance improvement is due to the use of speaker-independent language units as a means of data augmentation, thereby assisting the performance of the model with a small number of data samples.

[0263] Also, in both teacher-guided and speaker-independent timbre cloning, using speaker-independent language units helps the attention mechanism learn more robust alignment results, thereby improving the performance of the model with a small number of samples.

[0264] In the voice timbre conversion task, the performance of this method is better than that of a single-task VC model, and the improvement range is larger.

[0265] Therefore, overall, the multimodal timbre cloning system provided in this application scenario is superior to single-task TTS or VC models in various scenarios and has strong practical value.

[0266] Also, in an exemplary embodiment of the present disclosure, a voice generation device is further provided. FIG. 13 is a schematic diagram showing the configuration of the voice generation device. As shown in FIG. 13, the voice generation device 1300 may include a data acquisition module 1310, a vector determination module 1320, and a voice generation module 1330.

[0267] The data acquisition module 1310 is configured to acquire a voice feature vector of the voice to be processed, and input the voice feature vector into a voice generation model to obtain a language unit vector.

[0268] The vector determination module 1320 is configured to acquire a text feature vector, and determine a feature vector to be processed based on the text feature vector and the language unit vector.

[0269] The voice generation module 1330 is configured to input the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and input the acoustic feature vector into a vocoder to obtain a target voice corresponding to the voice to be processed or the text feature vector.

[0270] In an exemplary embodiment of the present invention, inputting the voice feature vector into a voice generation model to obtain a language unit vector includes: inputting the voice feature vector into the voice generation model such that the voice generation model outputs a voice encoding vector and a self-recovering voice; and performing loss calculation on the voice to be processed and the self-recovering voice to obtain a first loss value, and determining the voice encoding vector as the language unit vector based on the first loss value.

[0271] In an exemplary embodiment of the present invention, inputting the voice feature vector into the voice generation model such that the voice generation model outputs a voice encoding vector and a self-recovering voice includes: The aforementioned speech feature vector is input to a speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector. The speech encoding vector is quantized using the vector quantization module of the speech generation model to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained. The method includes obtaining self-reduced speech by nonlinearly transforming the speech quantization sequence and the speaker vector using the decoder module of the speech generation model.

[0272] In an exemplary embodiment of the present invention, obtaining the speaker vector corresponding to the audio to be processed is: The process involves obtaining a speaker identifier corresponding to the speech to be processed, determining the correspondence between the speaker identifier and the speaker vector, wherein the correspondence is determined based on the speech generation model. This includes querying the speaker vector corresponding to the speaker identifier based on the aforementioned correspondence.

[0273] In an exemplary embodiment of the present invention, obtaining a speech quantization sequence by quantizing the speech coding vector using the vector quantization module of the speech generation model is: This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm, based on the codebook in the vector quantization module of the speech generation model.

[0274] In an exemplary embodiment of the present invention, obtaining a speech quantization sequence by quantizing the speech coding vector using the nearest neighbor search algorithm is: The process involves updating the aforementioned codebook and obtaining the updated codebook, This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm based on the updated codebook.

[0275] In an exemplary embodiment of the present invention, updating the codebook and obtaining the updated codebook is: Obtain the codebook identifier of the codebook for each frame, compare the said codebook identifiers and obtain the comparison result, This includes merging the codebooks according to the comparison results and obtaining the updated codebooks.

[0276] In an exemplary embodiment of the present invention, inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector is, The process involves obtaining the acoustic vector to be processed from the feature vector to be processed, inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and having the sequence-to-sequence model output the processed acoustic vector. This includes performing a loss calculation on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and determining the processed acoustic vector as an acoustic feature vector based on the second loss value.

[0277] In an exemplary embodiment of the present invention, inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model causes the sequence-to-sequence model to output a processed acoustic vector, The process involves inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and obtaining a spatially encoded vector by nonlinear mapping the feature vector to be processed and the acoustic vector to be processed using the encoder module of the sequence-to-sequence model. The spatial coding vector and the speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is obtained. Aligning the alignment target vector and the speech feature sequence by the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and performing a non-linear mapping on the context representation vector by a decoder of the sequence-to-sequence model to obtain a processed acoustic vector, including:

[0278] In an exemplary embodiment of the present invention, determining a processing target feature vector based on the text feature vector and the language unit vector includes: Determining the text feature vector or the language unit vector as the processing target feature vector, or Adding the text feature vector and the language unit vector to obtain a processing target feature vector.

[0279] In an exemplary embodiment of the present invention, inputting the acoustic feature vector into a vocoder to obtain a target speech corresponding to the processing target speech or the text feature vector includes: Extracting the speech acoustic features of the acoustic feature vector through a post-processing network, and inputting the speech acoustic features into the vocoder, so that the vocoder outputs a pending speech corresponding to the processing target speech or the text feature vector, and Calculating a loss between the pending speech and the processing target speech to obtain a third loss value, and determining the pending speech as the target speech based on the third loss value.

[0280] In an exemplary embodiment of the present invention, calculating a loss between the pending speech and the processing target speech to obtain a third loss value includes: When the vocoder is an adversarial generation network, calculating a loss between the pending speech and the processing target speech to obtain an adversarial network loss value of the adversarial generation network, and This includes performing a loss calculation on the pending audio and the audio to be processed to obtain an audio feature loss value, and obtaining a third loss value by weighting and adding the adversarial network loss value and the audio feature loss value.

[0281] The specific details of the above-mentioned speech generation device 1300 are described in detail in the corresponding speech generation method, so the explanation is omitted here.

[0282] Although several modules or units of the voice generation device 1300 have been mentioned above, such division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more of the above modules or units may be embodied in a single module or unit. Conversely, the features and functions of a single module or unit may be embodied by multiple modules or units.

[0283] Furthermore, the present disclosure provides an electronic device capable of realizing the above method in an exemplary embodiment.

[0284] Next, an electronic device 1400 according to one embodiment of the present invention will be described with reference to Figure 14. The electronic device 1400 shown in Figure 14 is merely an example and does not impose any limitations on the function and scope of use of the embodiment of the present invention.

[0285] As shown in Figure 14, the electronic device 1400 is represented as a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to, the at least one processing unit 1410, the at least one storage unit 1420, a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410), and a display unit 1440.

[0286] Here, program code is stored in the storage unit, and the program code may be executed by the processing unit 1410, thereby performing the steps of various exemplary embodiments of the present invention as described in the “Exemplary Methods” section of this specification.

[0287] The aforementioned method, The process involves obtaining the speech feature vector of the audio to be processed, inputting the speech feature vector into a speech generation model to obtain a language unit vector, and The process involves obtaining a text feature vector, determining the feature vector to be processed based on the text feature vector and the language unit vector, This includes inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target speech corresponding to the speech to be processed or the text feature vector.

[0288] It is selectable to input the aforementioned speech feature vectors into a speech generation model to obtain language unit vectors. The speech feature vector is input to the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech. The process includes performing a loss calculation on the audio to be processed and the self-reduced audio to obtain a first loss value, and determining the speech coding vector as a language unit vector based on the first loss value.

[0289] It is selectable to input the speech feature vector to the speech generation model such that the speech generation model outputs a speech coding vector and a self-reduced speech, The aforementioned speech feature vector is input to a speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector. The speech encoding vector is quantized using the vector quantization module of the speech generation model to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained. The method includes obtaining self-reduced speech by nonlinearly transforming the speech quantization sequence and the speaker vector using the decoder module of the speech generation model.

[0290] It is selectable to obtain the speaker vector corresponding to the audio to be processed, The process involves obtaining a speaker identifier corresponding to the speech to be processed, determining the correspondence between the speaker identifier and the speaker vector, wherein the correspondence is determined based on the speech generation model. This includes querying the speaker vector corresponding to the speaker identifier based on the aforementioned correspondence.

[0291] It is selectable to obtain a speech quantization sequence by quantizing the speech coding vector using the vector quantization module of the speech generation model. This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm, based on the codebook in the vector quantization module of the speech generation model.

[0292] It is selectable to obtain a speech quantization sequence by quantizing the speech coding vector using the nearest neighbor search algorithm. The process involves updating the aforementioned codebook and obtaining the updated codebook, This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm based on the updated codebook.

[0293] It is selectable to update the aforementioned codebook and obtain the updated codebook. Obtain the codebook identifier of the codebook for each frame, compare the said codebook identifiers and obtain the comparison result, This includes merging the codebooks according to the comparison results and obtaining the updated codebooks.

[0294] It is selectable to input the aforementioned feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector. The process involves obtaining the acoustic vector to be processed from the feature vector to be processed, inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and having the sequence-to-sequence model output the processed acoustic vector. This includes performing a loss calculation on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and determining the processed acoustic vector as an acoustic feature vector based on the second loss value.

[0295] It is selectable that, by inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, the sequence-to-sequence model outputs the processed acoustic vector. The process involves inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and obtaining a spatially encoded vector by nonlinear mapping the feature vector to be processed and the acoustic vector to be processed using the encoder module of the sequence-to-sequence model. The spatial coding vector and the speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is obtained. This includes aligning the alignment target vector and the audio feature sequence using the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and performing a nonlinear mapping on the context representation vector using the decoder of the sequence-to-sequence model to obtain a processed acoustic vector.

[0296] Selectable, and determining the feature vector to be processed based on the text feature vector and the language unit vector, The text feature vector or the language The unit vector is determined as the feature vector to be processed, or This includes adding the text feature vector and the language unit vector to obtain the feature vector to be processed.

[0297] It is selectable to input the acoustic feature vector into a vocoder to obtain the target audio corresponding to the audio to be processed or the text feature vector, The process involves extracting the audio acoustic features of the acoustic feature vector via a post-processing network, inputting the audio acoustic features into a vocoder, and having the vocoder output pending audio corresponding to the audio to be processed or the text feature vector. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain a third loss value, and determining the pending audio as the target audio based on the third loss value.

[0298] It is selectable to perform loss calculations on the pending audio and the audio to be processed to obtain a third loss value. If the vocoder is a generative adversarial network, loss calculations are performed on the pending audio and the audio to be processed to obtain the generative adversarial network loss value of the generative adversarial network. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain an audio feature loss value, and obtaining a third loss value by weighting and adding the adversarial network loss value and the audio feature loss value.

[0299] The above method allows for the acquisition of speech and text feature vectors, enabling the reception of both speech and text as input. This integrates speech synthesis and speech-to-timbre conversion tasks, performing multimodal modeling and improving the performance of both tasks. Furthermore, for small amounts of data, it acquires speech and text feature vectors, providing multiple timbre cloning strategies, improving the effectiveness of timbre cloning with limited data, reducing the training difficulty and time for various models, and supporting timbre cloning methods in diverse application scenarios.

[0300] The memory unit 1420 may include a readable medium in the form of a volatile memory unit, such as a random access memory unit (RAM) 1421 and / or a cache memory unit 1422, and may further include a read-only memory unit (ROM) 1423.

[0301] The storage unit 1420 may further include a program / utility 1424 having a set (at least one) of program modules 1425, such program modules 1425 including, but not limited to, an operating system, one or more application programs, other program modules and program data, and each or any combination of these examples may include the implementation of a network environment.

[0302] Bus 1430 represents one or more of several types of bus structures, such as a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of the various bus architectures.

[0303] The electronic device 1400 may communicate with one or more external devices 1600 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices with which a user can interact, and / or any device (e.g., routers, modems, etc.) with which the electronic device 1400 can communicate with one or more other computing devices. Such communication can be performed via the input / output interface (I / O) 1450. The electronic device 1400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, e.g., the Internet) via the network adapter 1460. As shown in the figure, the network adapter 1460 communicates with other modules of the electronic device 1400 via the bus 1430. Although not shown in the figures, other hardware and / or software modules may be used in combination with the electronic device 1400, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0304] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by combining the necessary hardware with software. Accordingly, the technical solutions relating to the embodiments of this disclosure may be embodied in the form of a software product, which may be stored on a non-volatile storage medium (which may be a CD-ROM, U disk, portable hard disk, etc.) or on a network, and which includes several instructions to cause a computing device (which may be a personal computer, server, terminal device, or network device, etc.) to perform the method relating to the embodiments of this disclosure.

[0305] In exemplary embodiments of this disclosure, a computer-readable storage medium is further provided which stores a program product capable of implementing the above methods of this specification. In some possible embodiments, each aspect of the present invention includes program code, and if the program product is executed on a terminal device, the program code may be implemented in the form of a program product used to cause the terminal device to perform steps according to various exemplary embodiments of the present invention as described in the “exemplary methods” portion of this specification.

[0306] The aforementioned method, The process involves obtaining the speech feature vector of the audio to be processed, inputting the speech feature vector into a speech generation model to obtain a language unit vector, and The process involves obtaining a text feature vector, determining the feature vector to be processed based on the text feature vector and the language unit vector, This includes inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target speech corresponding to the speech to be processed or the text feature vector.

[0307] It is selectable to input the aforementioned speech feature vectors into a speech generation model to obtain language unit vectors. The speech feature vector is input to the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech. The process includes performing a loss calculation on the audio to be processed and the self-reduced audio to obtain a first loss value, and determining the speech coding vector as a language unit vector based on the first loss value.

[0308] It is selectable to input the speech feature vector to the speech generation model such that the speech generation model outputs a speech coding vector and a self-reduced speech, The aforementioned speech feature vector is input to a speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector. The speech encoding vector is quantized using the vector quantization module of the speech generation model to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained. The method includes obtaining self-reduced speech by nonlinearly transforming the speech quantization sequence and the speaker vector using the decoder module of the speech generation model.

[0309] It is selectable to obtain the speaker vector corresponding to the audio to be processed, The process involves obtaining a speaker identifier corresponding to the speech to be processed, determining the correspondence between the speaker identifier and the speaker vector, wherein the correspondence is determined based on the speech generation model. This includes querying the speaker vector corresponding to the speaker identifier based on the aforementioned correspondence.

[0310] It is selectable to obtain a speech quantization sequence by quantizing the speech coding vector using the vector quantization module of the speech generation model. This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm, based on the codebook in the vector quantization module of the speech generation model.

[0311] It is selectable to obtain a speech quantization sequence by quantizing the speech coding vector using the nearest neighbor search algorithm. The process involves updating the aforementioned codebook and obtaining the updated codebook, This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm based on the updated codebook.

[0312] It is selectable to update the aforementioned codebook and obtain the updated codebook. Obtain the codebook identifier of the codebook for each frame, compare the said codebook identifiers and obtain the comparison result, This includes merging the codebooks according to the comparison results and obtaining the updated codebooks.

[0313] It is selectable to input the aforementioned feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector. The process involves obtaining the acoustic vector to be processed from the feature vector to be processed, inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and having the sequence-to-sequence model output the processed acoustic vector. This includes performing a loss calculation on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and determining the processed acoustic vector as an acoustic feature vector based on the second loss value.

[0314] It is selectable that, by inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, the sequence-to-sequence model outputs the processed acoustic vector. The process involves inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and obtaining a spatially encoded vector by nonlinear mapping the feature vector to be processed and the acoustic vector to be processed using the encoder module of the sequence-to-sequence model. The spatial coding vector and the speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is obtained. This includes aligning the alignment target vector and the audio feature sequence using the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and performing a nonlinear mapping on the context representation vector using the decoder of the sequence-to-sequence model to obtain a processed acoustic vector.

[0315] Selectable, and determining the feature vector to be processed based on the text feature vector and the language unit vector, The text feature vector or the language The unit vector is determined as the feature vector to be processed, or This includes adding the text feature vector and the language unit vector to obtain the feature vector to be processed.

[0316] It is selectable to input the acoustic feature vector into a vocoder to obtain the target audio corresponding to the audio to be processed or the text feature vector, The process involves extracting the audio acoustic features of the acoustic feature vector via a post-processing network, inputting the audio acoustic features into a vocoder, and having the vocoder output pending audio corresponding to the audio to be processed or the text feature vector. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain a third loss value, and determining the pending audio as the target audio based on the third loss value.

[0317] It is selectable to perform loss calculations on the pending audio and the audio to be processed to obtain a third loss value. If the vocoder is a generative adversarial network, loss calculations are performed on the pending audio and the audio to be processed to obtain the generative adversarial network loss value of the generative adversarial network. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain an audio feature loss value, and obtaining a third loss value by weighting and adding the adversarial network loss value and the audio feature loss value.

[0318] The above method allows for the acquisition of speech and text feature vectors, enabling the reception of both speech and text as input. This integrates speech synthesis and speech-to-timbre conversion tasks, performing multimodal modeling and improving the performance of both tasks. Furthermore, for small amounts of data, it acquires speech and text feature vectors, providing multiple timbre cloning strategies, improving the effectiveness of timbre cloning with limited data, reducing the training difficulty and time for various models, and supporting timbre cloning methods in diverse application scenarios.

[0319] As shown in Figure 15, the program product 1500 for implementing the above method according to an embodiment of the present invention employs a portable compact disk read-only memory (CD-ROM) containing program code and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this specification, the readable storage medium may be any tangible medium containing the program, and the program may be instructed to be used with or in combination with a system, apparatus, or device.

[0320] The program product may use any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples of readable storage media (a non-exclusive list) include: an electrical connection with one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the above.

[0321] A computer-readable signaling medium may include propagated data signals containing readable program code in the baseband or as part of a carrier wave. Such propagated data signals may include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof, and can employ a variety of forms. The computer-readable signaling medium may be any other-readable medium besides a computer-readable storage medium, and such medium may transmit, propagate, or transmit a program used in conjunction with an instruction execution system, apparatus, or device.

[0322] Program code contained in a readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, fiber optic cable, RF, or any suitable combination thereof.

[0323] The program code for performing the operations of the present invention can be written using one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as the C programming language or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the remote computing device scenario, the remote computing device can connect to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device (for example, via the Internet using an Internet service provider).

[0324] A person skilled in the art will readily conceive of other embodiments of the Disclosure after considering the specification and practicing the inventions disclosed herein. This Disclosure is intended to cover any variations, uses, or adaptive changes of the Disclosure, which will conform to the general principles of the Disclosure and include common or conventional art means that are not disclosed herein. This Specification and the Examples are to be considered merely illustrative, and the true scope and spirit of the Disclosure are indicated by the Claims.

Claims

1. The process involves obtaining the speech feature vector of the audio to be processed, inputting the speech feature vector into a speech generation model to obtain a language unit vector, and The process involves obtaining a text feature vector, determining a feature vector to be processed based on the text feature vector and the language unit vector, and deciding to obtain the feature vector to be processed by adding the text feature vector and the language unit vector. This includes inputting the feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and inputting the acoustic feature vector into a vocoder to obtain a target speech corresponding to the audio to be processed or the text feature vector, A method for generating speech characterized by the following features.

2. Inputting the aforementioned speech feature vectors into a speech generation model to obtain language unit vectors is, The speech feature vector is input to the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech. This includes performing a loss calculation on the audio to be processed and the self-reduced audio to obtain a first loss value, and determining the speech coding vector as a language unit vector based on the first loss value, The speech generation method according to feature 1.

3. Inputting the speech feature vector into the speech generation model so that the speech generation model outputs a speech coding vector and a self-reduced speech vector is: The aforementioned speech feature vector is input to a speech generation model, and the speech feature vector is nonlinearly transformed by the encoder module of the speech generation model to obtain a speech encoded vector. The speech encoding vector is quantized using the vector quantization module of the speech generation model to obtain a speech quantization sequence, and a speaker vector corresponding to the speech to be processed is obtained. The decoder module of the speech generation model obtains self-reduced speech by nonlinearly transforming the speech quantization sequence and the speaker vector, The speech generation method according to feature 2.

4. Obtaining the speaker vector corresponding to the audio to be processed means The process involves obtaining a speaker identifier corresponding to the speech to be processed, determining the correspondence between the speaker identifier and the speaker vector, wherein the correspondence is determined based on the speech generation model. This includes querying the speaker vector corresponding to the speaker identifier based on the aforementioned correspondence, The speech generation method according to feature 3.

5. Obtaining a speech quantization sequence by quantizing the speech coding vector using the vector quantization module of the speech generation model is: This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor (KNN, K-NearestNeighbor) search algorithm based on the codebook in the vector quantization module of the speech generation model. The speech generation method according to feature 3.

6. Obtaining a speech quantized sequence by quantizing the speech coding vector using the nearest neighbor search algorithm is: The process involves updating the aforementioned codebook and obtaining the updated codebook, This includes obtaining a speech quantization sequence by quantizing the speech coding vector using a nearest neighbor search algorithm based on the updated codebook, The speech generation method according to feature 5.

7. Updating the aforementioned codebook and obtaining the updated codebook is, Obtain the codebook identifier of the codebook for each frame, compare the said codebook identifiers and obtain the comparison result, This includes merging the codebooks according to the comparison results and obtaining the updated codebooks, The speech generation method according to feature 6.

8. Inputting the aforementioned feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector is, The process involves obtaining the acoustic vector to be processed from the feature vector to be processed, inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and having the sequence-to-sequence model output the processed acoustic vector. This includes performing a loss calculation on the acoustic vector to be processed and the processed acoustic vector to obtain a second loss value, and determining the processed acoustic vector as an acoustic feature vector based on the second loss value. The speech generation method according to feature 3.

9. By inputting the aforementioned feature vector to be processed and the aforementioned acoustic vector to be processed into a sequence-to-sequence model, the sequence-to-sequence model outputs the processed acoustic vector. The process involves inputting the feature vector to be processed and the acoustic vector to be processed into a sequence-to-sequence model, and obtaining a spatially encoded vector by nonlinear mapping the feature vector to be processed and the acoustic vector to be processed using the encoder module of the sequence-to-sequence model. The spatial coding vector and the speaker vector are added together to obtain the alignment target vector, and the speech feature sequence is obtained. This includes aligning the alignment target vector and the audio feature sequence using the attention mechanism of the sequence-to-sequence model to obtain a context representation vector, and performing a nonlinear mapping on the context representation vector using the decoder of the sequence-to-sequence model to obtain a processed acoustic vector. The voice generation method according to feature 8.

10. Inputting the aforementioned acoustic feature vector into a vocoder to obtain the target audio corresponding to the audio to be processed or the text feature vector is: The process involves extracting the audio acoustic features of the acoustic feature vector via a post-processing network, inputting the audio acoustic features into a vocoder, and having the vocoder output pending audio corresponding to the audio to be processed or the text feature vector. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain a third loss value, and determining the pending audio as the target audio based on the third loss value. The speech generation method according to feature 1.

11. Performing loss calculations on the pending audio and the audio to be processed to obtain a third loss value is: If the vocoder is a generative adversarial network, loss calculations are performed on the pending audio and the audio to be processed to obtain the generative adversarial network loss value of the generative adversarial network. This includes performing a loss calculation on the pending audio and the audio to be processed to obtain an audio feature loss value, and then weighting and adding the adversarial network loss value and the audio feature loss value to obtain a third loss value. The voice generation method according to feature 10.

12. A data acquisition module configured to acquire speech feature vectors of the speech to be processed, input the speech feature vectors into a speech generation model to acquire language unit vectors, A vector determination module configured to acquire a text feature vector and determine a feature vector to be processed based on the text feature vector and the language unit vector, comprising: a vector determination module that determines to acquire a feature vector to be processed by adding the text feature vector and the language unit vector; The system includes a speech generation module configured to input the aforementioned feature vector to be processed into a sequence-to-sequence model to obtain an acoustic feature vector, and to input the acoustic feature vector into a vocoder to obtain the aforementioned speech to be processed or a target speech corresponding to the aforementioned text feature vector. A voice generation device characterized by the following features.

13. A computer-readable storage medium in which a computer program is stored, wherein the computer program, when executed by a processor, realizes the speech generation method described in any one of claims 1-11. A computer-readable storage medium characterized by the following features.

14. An electronic device comprising a processor and memory, The memory is configured to store executable instructions of the processor, The processor is configured to execute the speech generation method described in any one of claims 1 to 11 by executing the executable instructions. An electronic device characterized by the following features.

Images