Methods and apparatus for generating multi-style speech

Abstract

This disclosure provides a method and apparatus for generating multi-style speech. The method first obtains the style features of audio samples, then obtains the feature types and loss values, updates the model parameters using the loss values, and uses adaptive feature transfer technology to transfer style features to achieve style speech synthesis. While ensuring model performance, it effectively reduces the complexity of model training and the amount of data used during model training. In speech synthesis, different styles are modeled, and multi-speaker speech synthesis technology is used to synthesize different styles for the same sentence. By switching different second audio as input, multiple style transfers can be achieved, thereby transferring different style features to any unlabeled speaker. This improves the naturalness and expressiveness of synthesized speech without reducing the quality of speech synthesis.

Description

Technical Field

[0001] This disclosure relates to the field of speech synthesis technology, and in particular to a method and apparatus for generating multi-style speech. Background Technology

[0002] Currently, in TTS (Text to Speech) tasks, existing models for modeling audio styles only consider using a single audio file to learn style representations during training. However, in real-world scenarios, different styles overlap, meaning that two different styles may share commonalities. Therefore, considering all styles during training becomes particularly important for the model to learn different style representations.

[0003] Existing work requires a large amount of style-labeled data to help the model learn appropriate style representations. The amount of data required for training is large, which is time-consuming and labor-intensive. Furthermore, the uncertainty in the labeling makes model training difficult and increases the complexity of the model. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this disclosure provides a method and apparatus for generating multi-style speech.

[0005] The first aspect of this disclosure proposes a method for generating multi-style speech, comprising: acquiring style features of all audio samples in a training set through a style feature extraction module; clustering all acquired style features to obtain multiple style feature types; obtaining semantic features of the audio samples through a neural network vocoder, and obtaining a first loss value based on the semantic features and the style features; determining the distance between the style features corresponding to the audio samples and the multiple style feature types, and calculating a distance loss value corresponding to the distance based on a contrastive loss function; updating the parameters of a style transfer module based on the comprehensive loss value to reduce the distance until a convergence condition is reached, thereby obtaining a trained speech style transfer model, wherein the comprehensive loss value includes the first loss value and the distance loss value, the speech style transfer model includes the neural network vocoder and the style transfer module, and the style transfer module includes the style feature extraction module; and inputting a first audio into the neural network vocoder and inputting a second audio into the style feature extraction module to obtain synthesized audio, wherein the text content of the synthesized audio corresponds to the first audio and the audio style corresponds to the second audio.

[0006] According to one embodiment of this disclosure, obtaining style features of all audio samples in the training set through a style feature extraction module includes: obtaining the spectrum of all audio samples in the training set, and extracting style features from the spectrum through the style feature extraction module.

[0007] According to one embodiment of this disclosure, before clustering all acquired style features, the method further includes: denoising the style features extracted by the style feature extraction module using a multilayer perceptron.

[0008] According to one embodiment of this disclosure, obtaining the semantic features of the audio sample through a neural network vocoder includes: inputting the spectrum of the audio sample into the neural network vocoder to obtain the semantic features.

[0009] According to one embodiment of this disclosure, inputting the spectrum of the audio sample into a neural network vocoder to obtain semantic features includes: performing a one-dimensional convolution on the spectrum of the audio sample to obtain a first convolution result; and performing multiple anti-aliasing multi-cycle synthesis operations on the first convolution result to obtain semantic features, wherein each anti-aliasing multi-cycle synthesis operation includes sequentially performing a one-dimensional transposed convolution and an anti-aliasing multi-cycle synthesis operation.

[0010] According to one embodiment of this disclosure, obtaining a first loss value based on the semantic features and the style features includes: performing matrix addition on the semantic features and the style features to obtain a feature to be processed; and performing nonlinear processing on the feature to be processed to obtain the first loss value.

[0011] According to one embodiment of this disclosure, determining the distance between the style feature corresponding to the audio sample and the multiple style feature types, and calculating the distance loss value corresponding to the distance based on the contrast loss function, includes: for each style feature, determining the style feature type closest to the style feature from the multiple style feature types as the target type; and inputting the distance between each style feature and the corresponding target type into the contrast loss function to obtain the distance loss value.

[0012] According to one embodiment of this disclosure, the style transfer module further includes a personality feature learning module, and the method further includes: inputting the unique identifier of the audio sample into the personality feature learning module for mapping, and inputting the mapping result and the corresponding style feature into the expected minimization loss function to obtain an expected loss value; correspondingly, the first loss value is obtained simultaneously based on the semantic feature, the style feature and the mapping result; correspondingly, the comprehensive loss value also includes the mapping result, and when updating the parameters of the style transfer module based on the comprehensive loss value, the personality features of the corresponding audio sample are learned by updating the parameters of the personality feature learning module to reduce the personality features in the style features, wherein the personality features include age features and / or gender features and / or timbre features.

[0013] According to one embodiment of this disclosure, the text content of the first audio is the same as the text content of the audio samples in the training set. When the first audio is input into the neural network vocoder, the unique identifier of the first audio is also input into the individual feature learning module to reduce the individual features of the second audio present in the obtained synthesized audio.

[0014] A second aspect of this disclosure provides an apparatus for generating multi-style speech, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the multi-style speech generation method described in any of the above embodiments. Attached Figure Description

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 This is a flowchart illustrating a method for generating multi-style speech according to one embodiment of the present disclosure.

[0017] Figure 2 This is a schematic diagram of the training process of a speech style transfer model according to one embodiment of the present disclosure.

[0018] Figure 3 This is a schematic diagram illustrating the usage process of a speech style transfer model according to one embodiment of the present disclosure.

[0019] Figure 4 This is a schematic diagram of a multi-style speech generation apparatus employing a hardware implementation of a processing system according to one embodiment of the present disclosure. Detailed Implementation

[0020] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0021] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0023] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0024] The method and apparatus for generating multi-style speech disclosed herein are described below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a method for generating multi-style speech according to one embodiment of this disclosure. See also... Figure 1 The multi-style speech generation method M10 of this embodiment may include the following steps S100, S200, S300, S400 and S500.

[0026] S100 obtains the style features of all audio samples in the training set through the style feature extraction module.

[0027] S200 clusters all acquired style features to obtain multiple style feature types.

[0028] S300 obtains the semantic features of audio samples through a neural network vocoder, and obtains the first loss value based on the semantic features and style features.

[0029] S400 determines the distance between the style features corresponding to the audio samples and various style feature types, and calculates the distance loss value corresponding to the distance based on the contrast loss function.

[0030] S500 updates the parameters of the style transfer module based on the comprehensive loss value to reduce the distance until the convergence condition is met, thus obtaining a trained speech style transfer model. The comprehensive loss value includes the first loss value and the distance loss value. The speech style transfer model includes a neural network vocoder and a style transfer module. The style transfer module includes a style feature extraction module.

[0031] S600, the first audio is input into the neural network vocoder, and the second audio is input into the style feature extraction module to obtain the synthesized audio. The text content of the synthesized audio corresponds to the first audio and the audio style corresponds to the second audio.

[0032] The multi-style speech generation method proposed in the embodiments of this disclosure employs adaptive feature transfer technology to transfer style features, thereby achieving style speech synthesis. While ensuring model performance, it effectively reduces the complexity of model training and the amount of data used during model training. Different styles are modeled in speech synthesis, and multi-speaker speech synthesis technology is used to synthesize different styles for the same sentence. By switching different second audio as input, multiple style transfers can be achieved, thereby transferring different style features to any unlabeled speaker. This improves the naturalness and expressiveness of synthesized speech without reducing the quality of speech synthesis.

[0033] The training set may contain, for example, 250 training corpora from different speakers; these training corpora are the audio samples. Audio exhibits a wide variety of emotional styles; different audio recordings may convey different emotions—some are gentle and lovely, some are serious and earnest, some are magnetic, and some are sad and sorrowful. Style features are the characteristic data that can characterize the emotional style of the audio. For example, the 250 corpora in the training set contain approximately 23 different styles.

[0034] Figure 2 This is a schematic diagram illustrating the training process of a speech style transfer model according to one embodiment of this disclosure. (See also...) Figure 2 In step S100, the method of obtaining the style features of all audio samples in the training set through the style feature extraction module may include the following steps: obtaining the spectrum of all audio samples in the training set, and extracting the style features from the spectrum through the style feature extraction module.

[0035] The spectrum can be a Mel spectrogram. The Mel spectrogram is obtained through Mel filtering. The style feature extraction module can be a GST module. The GST (Global Style Token) module can extract style features from the spectrum. Style features can be represented as tensors. By learning and updating the parameters of the GST module, the GST module can learn the emotional style representation of the spectrum (e.g., happy, sad, cute, etc.).

[0036] After obtaining the style features of all 250 audio samples, the style features extracted by the style feature extraction module (GST module) can be denoised using a Multilayer Perceptron (MLP) before clustering the 250 style features. The MLP can filter the style features extracted by the GST module, removing useless information or noise. At this point, the style features are coarse-grained because they include not only the speaker's emotional style but also their personality traits. Personality traits can include the speaker's timbre, age, etc., such as a high-pitched girl's voice or a deep, hoarse adult man's voice. These are unrelated to the speaker's emotional style and belong to the personality traits within the style features.

[0037] In step S200, the clustering method can be unsupervised clustering, such as using the K-means clustering algorithm. From the obtained 250 coarse-grained style representations, more generalized representations are extracted; for example, style features related to sadness, grief, and breakdown are grouped into the sadness style feature category. The clustering result could be, for example, 23 clusters. Figure 2 The clustering results are illustrated using four small rectangles with different filling patterns.

[0038] After clustering, the Mel spectrograms of the 250 audio samples are sequentially input into the neural network vocoder in batches to train the GST module. Assume there are 5 batches, each containing 50 spectrograms. Taking the training of one batch as an example, the 50 spectrograms are sequentially input into the neural network vocoder.

[0039] Neural vocoders can employ the BigVGAN model. BigVGAN is a pre-trained vocoder that introduces periodic nonlinearity and anti-aliasing representations into the generator, providing the necessary inductive bias for waveform synthesis and improving audio quality. It can extract general features from audio for style transfer. Since BigVGAN is a pre-trained model, its parameters are not updated during model training and parameter updates.

[0040] BigVGAN can extract semantic features from audio samples. Semantic features refer to features related to the text and prosody of the audio samples. In step S300, the method of obtaining the semantic features of the audio samples through a neural network vocoder may include the following steps: inputting the spectrum of the audio samples into the neural network vocoder to obtain the semantic features.

[0041] For example, the method of inputting the spectrum of an audio sample into a neural network vocoder to obtain semantic features may include the following steps: performing a one-dimensional convolution on the spectrum of the audio sample to obtain a first convolution result; and performing multiple anti-aliasing multi-cycle synthesis operations on the first convolution result to obtain semantic features, wherein each anti-aliasing multi-cycle synthesis operation includes performing a one-dimensional transposed convolution and an anti-aliasing multi-cycle synthesis operation in sequence.

[0042] For the current batch of 50 spectra, after inputting them into the BigVGAN model, a one-dimensional convolution (Conv1d) is performed first, followed by N anti-aliased multi-periodicity (AMP) operations to deepen the neural network and capture long-term relationships between features. Specifically, the input to the first AMP operation is the output of the one-dimensional convolution. This output is then subjected to a one-dimensional transposed convolution (TransposedConv1d), followed by AMP operations on the transposed convolution result. The input to the second through Nth AMP operations is the output of the previous AMP operation. The output of the Nth AMP operation is the semantic feature.

[0043] After obtaining the semantic features of this batch, the method of obtaining the first loss value based on the semantic features and style features in step S300 may include the following steps: adding the semantic features and style features into a matrix to obtain the features to be processed; and performing nonlinear processing on the features to be processed to obtain the first loss value.

[0044] Semantic features and style features are both two-dimensional tensors; adding them together yields a new two-dimensional tensor. Figure 2 In this diagram, "⊕" represents matrix addition. During the nonlinearization process, linear features are first converted to nonlinear features using a first activation function. This first activation function can be the Snake1d function. The Snake1d function's input and output are both tensors; for a new two-dimensional tensor x, it can be transformed into a representation exhibiting periodic changes. The result after nonlinearization is then subjected to a one-dimensional convolution (Conv1d), followed by numerical mapping using a second activation function. The second activation function can be the Tanh activation function, mapping the values ​​to the interval [-1, 1] to obtain the first loss value.

[0045] For the 50 style features in this batch, since 23 cluster centers have already been obtained through clustering in step S200, the distance between each style feature and each cluster center in the 23 clusters can be obtained. The same principle applies to subsequent batches; the distance between each style feature and each cluster center can be obtained. Distance refers to the distance between two tensors in the same space.

[0046] For example, step S400 may include the following steps: for each style feature, determining the style feature type that is closest to the style feature from a variety of style feature types as the target type; and inputting the distance between each style feature and the corresponding target type into the contrast loss function to obtain a distance loss value.

[0047] Assuming 50 style features are numbered c1 to c50, taking c1 as an example, the distance between c1 and the 23 cluster centers is calculable. Among these 23 cluster centers, there will be one cluster center that is closest to c1; this cluster center is the target type of c1, and the distance between c1 and the target type is the target distance value of c1. The magnitude of the distance value represents the similarity between two tensors; the smaller the distance, the more similar they are. The 23 cluster centers, as the clustering results, can be stored in a table. The size of the table is the product of the number of hidden features and 23. By inputting the style feature tensors into the table, the closest cluster center can be found. The same logic applies to c2 to c50, yielding the corresponding target types and target distance values. Inputting the 50 target distance values ​​into the loss function yields a distance loss value corresponding to this batch.

[0048] It is understandable that the style features used for distance calculation in step S400 can directly use the style features obtained by the GST module in step S100, or the 50 spectra from this batch can be re-inputted into the GST module and the multilayer perceptron (MLP) to obtain style features again. The style features obtained again are completely consistent with the style features obtained in step S100. This embodiment takes the direct use of the style features obtained in step S100 as an example.

[0049] It should be noted that steps S400 and S300 can start executing simultaneously, or they can be executed earlier or later than step S300. There is no dependency between steps S300 and S400, so the execution order is not restricted.

[0050] The speech style transfer model can consist of only BigVGAN and GST modules. After obtaining the distance loss value, the distance loss value and the first loss value obtained in step S300 are used as the comprehensive loss value to adjust and update the parameters of the GST module, thereby narrowing the distance between style features and the nearest cluster centers, making the two tensors more similar, and enabling the GST module to learn the style representation in the spectrum of this batch. This process is then iterated repeatedly until the loss value is below a preset threshold, or the preset number of iterations is reached, or other preset convergence conditions are met, thus completing the training of this batch. If there are training samples in the training set that have not yet been trained, the next batch of training can begin, until all batches of training are completed and a trained speech style transfer model is obtained. It should be noted that the loss value does not update the parameters of the BigVGAN neural network vocoder.

[0051] Figure 3 This is a schematic diagram illustrating the usage flow of a speech style transfer model according to one embodiment of this disclosure. (See also...) Figure 3 After training the speech style transfer model, in practical applications, the audio spectrum of the style to be transferred can be input into BigVGAN as the first audio spectrum Spectrogram1, and the audio spectrum providing the desired style can be input into the GST module as the second audio spectrum Spectrogram2. The trained GST module can correctly extract the style of Spectrogram2. BigVGAN then concatenates the style of Spectrogram2 onto the general features of Spectrogram1, achieving style transfer and obtaining a synthesized audio Wave that speaks Spectrogram1 in the style of Spectrogram2. For example, if Spectrogram1 corresponds to an audio clip with a sad tone and its text content is A, and Spectrogram2 corresponds to an audio clip with an angry tone, then the resulting synthesized audio Wave is an audio Wave that speaks text A in an angry tone.

[0052] For example, the style transfer module may also include a personality feature learning module. The multi-style speech generation method M10 may also include the following steps: inputting the unique identifier of the audio sample into the personality feature learning module for mapping, and inputting the mapping result and the corresponding style features into the expected minimization loss function to obtain the expected loss value.

[0053] Accordingly, the first loss value can be obtained simultaneously based on semantic features, style features, and mapping results.

[0054] Accordingly, the overall loss value may also include the mapping result. When updating the parameters of the style transfer module based on the overall loss value, the individual characteristics of the corresponding audio samples can be learned by updating the parameters of the individual characteristic learning module to reduce the individual characteristics in the style features. The individual characteristics include age features and / or gender features and / or timbre features.

[0055] See Figure 2 The individual feature learning module can employ the Embedding module. The Embedding module can use the built-in torch method `nn.Embedding`, which can densify sparse tensors. The unique identifier can be the ID of the audio sample; the Embedding module maps the feature ID from a high-dimensional sparse space to a low-dimensional dense space. When inputting 50 spectra into BigVGAN in step S300, the speaker IDs of these 50 spectra can be simultaneously input into the Embedding module. Assuming there are 250 training samples, the Embedding module can generate a two-dimensional tensor table of size 250*H and fill it with random numbers, where H is the number of hidden features, which can be set to 192. If the 50 spectrum IDs in this batch correspond to the 1st to 50th index positions in the two-dimensional table, then when the nth ID is input, the Embedding module will output a matrix of size [1, H] at the nth index position. This [1, H] matrix is ​​the mapping result. After inputting the SpeakerIDs of 50 spectra, the Embedding module will output the mapping result of the 50 index positions.

[0056] The mapping result serves two purposes. First, in step S300, the mapping result, semantic features, and style features are used together to calculate the first loss value. When calculating the first loss value, an aggregation operation can be performed on the style features and the mapping result. This aggregation operation can be addition, multiplication, finding the maximum or minimum value, or averaging; this implementation uses addition. Then, the aggregated result is added to the semantic features.

[0057] The second function is to perform expectation minimization to learn the individual characteristics of the corresponding audio samples, thereby reducing the individual characteristics in the style features. Specifically, before step S500, after obtaining the mapping results and style features, the mapping results and style features can be input into the expectation minimization loss function to obtain the expected loss value. When updating the parameters of the style transfer module, updates are performed simultaneously based on the first loss value, the distance loss value, and the expected loss value. At this time, the first loss value is calculated based on the mapping results, semantic features, and style features. In addition to updating the parameters of the GST module in the style transfer module to reduce the distance between the style features and the corresponding cluster centers, the tensor parameters of the Embedding module in the style transfer module are also updated to learn the individual characteristics (age, gender, timbre, etc.) of the spectral Mel Spectrogram.

[0058] The ID input to the Embedding module corresponds to the position of the parameter updated by the Embedding module during the current batch of training. Therefore, when updating the parameters of the Embedding module, the updated parameters are the parameters at the corresponding 50 positions in the tensor table, i.e., updating 50 [1,H] matrices. If there are 250 samples in the training set, then after all training samples in the training set have completed training, the parameters at all positions in the tensor table of the Embedding module have also been updated, resulting in a trained Embedding module.

[0059] The Embedding module learns individual characteristics and then uses an expectation-minimization loss function to ensure that the learned style representation contains only the speaker's style characteristics and not their personal features. Specifically, the expectation-minimization loss function can employ negative Kullback-Leibler divergence (KLD) to minimize the similarity between two tensors, preventing their individual characteristics from becoming too similar.

[0060] The text content of the first audio is the same as the text content of the audio samples in the training set. When the first audio is input into the neural network vocoder, the unique identifier of the first audio is also input into the individual feature learning module to reduce the individual features of the second audio in the resulting synthesized audio.

[0061] After training is complete, when using the model for style transfer, the input data to the model consists of three parts: Spectrogram1, Spectrogram2, and the ID of Spectrogram1. Spectrogram1 can be a training sample from the training set. The synthesized audio Wave output by the model weakens or even removes the individual characteristics of Spectrogram2, striving to ensure that the transferred content only contains stylistic features. For example, if Spectrogram1 corresponds to an audio recording of a sad adult man, and Spectrogram2 corresponds to an audio recording of an angry young woman, then the synthesized audio Wave will have an angry tone, but age and gender characteristics will still retain the timbre of an adult man without completely altering the timbre of a young woman.

[0062] It is understandable that if the input data to the model only contains spectrum Spectrogram1 and spectrum Spectrogram2 without ID information, it can still play a certain role in stripping individual features, because the learning and training of individual feature stripping has already been carried out during the training phase. However, the synthesized audio obtained by inputting the ID will be better than that obtained by not inputting the ID.

[0063] The method is evaluated using the Mean Opinion Score (MOS) and confusion matrix. The audio synthesized by the speech generation method of this embodiment is compared with that of existing methods and user ratings are performed. Accuracy, F1 score and other indicators are calculated for three different tones. The results show that the method provided by this embodiment has a significant effect on the style of synthesized audio with less data, and the naturalness MOS score is significantly higher than that of audio generated by existing methods without reducing the quality of speech synthesis.

[0064] Figure 4 This is a schematic diagram of a multi-style speech generation apparatus employing a hardware implementation of a processing system according to one embodiment of the present disclosure. (See also...) Figure 4 The multi-style speech generation apparatus 1000 of this embodiment may include a memory 1300 and a processor 1200. The memory 1300 stores execution instructions, and the processor 1200 executes the execution instructions stored in the memory 1300, causing the processor 1200 to execute the multi-style speech generation method of any of the above embodiments.

[0065] The device 1000 may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0066] For example, the multi-style speech generation device 1000 may include a style feature extraction module 1002, a style feature clustering module 1004, a first loss value acquisition module 1006, a distance loss acquisition module 1008, a model training module 1010, and an audio synthesis module 1012.

[0067] The style feature extraction module 1002 is used to acquire style features of all audio samples in the training set. The style feature extraction module 1002 may acquire style features of all audio samples in the training set by acquiring the spectrum of all audio samples in the training set and extracting style features from the spectrum. Before clustering all acquired style features, the style feature extraction module 1002 may also denoise the extracted style features using a multilayer perceptron.

[0068] The style feature clustering module 1004 is used to cluster all the acquired style features to obtain multiple style feature types.

[0069] The first loss value acquisition module 1006 is used to obtain the semantic features of the audio sample through a neural network vocoder, and to obtain a first loss value based on the semantic features and style features. The method by which the first loss value acquisition module 1006 obtains the semantic features of the audio sample through the neural network vocoder may include: inputting the spectrum of the audio sample into the neural network vocoder to obtain the semantic features. The method by which the first loss value acquisition module 1006 inputs the spectrum of the audio sample into the neural network vocoder to obtain the semantic features may include: performing a one-dimensional convolution on the spectrum of the audio sample to obtain a first convolution result; and performing multiple anti-aliasing multi-period synthesis operations on the first convolution result to obtain the semantic features, wherein each anti-aliasing multi-period synthesis operation includes sequentially performing a one-dimensional transposed convolution and an anti-aliasing multi-period synthesis operation. The method by which the first loss value acquisition module 1006 obtains the first loss value based on the semantic features and style features may include: adding the semantic features and style features as matrices to obtain the feature to be processed; and performing non-linear processing on the feature to be processed to obtain the first loss value.

[0070] The distance loss acquisition module 1008 is used to determine the distance between the style features corresponding to the audio samples and various style feature types, and to calculate the distance loss value corresponding to the distance based on the contrast loss function. Specifically, for each style feature, the distance loss acquisition module 1008 can determine the style feature type that is closest to the style feature from the various style feature types as the target type; and input the distance between each style feature and the corresponding target type into the contrast loss function to obtain the distance loss value.

[0071] The model training module 1010 is used to update the parameters of the style transfer module based on the comprehensive loss value to reduce the distance until the convergence condition is met, thereby obtaining the trained speech style transfer model. The comprehensive loss value includes the first loss value and the distance loss value. The speech style transfer model includes a neural network vocoder and a style transfer module. The style transfer module includes a style feature extraction module 1002.

[0072] The style transfer module may also include a personality feature learning module. The multi-style speech generation device 1000 may further include an expected loss value acquisition module, which can input the unique identifier of the audio sample into the personality feature learning module for mapping, and input the mapping result and the corresponding style features into the expected loss function to obtain the expected loss value. Accordingly, the first loss value can be obtained simultaneously based on semantic features, style features, and the mapping result. Correspondingly, the comprehensive loss value may also include the mapping result. When updating the parameters of the style transfer module based on the comprehensive loss value, the model training module 1010 learns the personality features of the corresponding audio sample by updating the parameters of the personality feature learning module to reduce the personality features in the style features. Personality features include age features and / or gender features and / or timbre features.

[0073] The audio synthesis module 1012 is used to input a first audio file into a neural network vocoder and input a second audio file into a style feature extraction module 1002 to obtain a synthesized audio file. The text content of the synthesized audio file corresponds to the first audio file, and the audio style corresponds to the second audio file. The text content of the first audio file can be the same as the text content of audio samples in the training set. When inputting the first audio file into the neural network vocoder, the audio synthesis module 1012 can also input the unique identifier of the first audio file into the individual feature learning module to reduce the individual features of the second audio file present in the obtained synthesized audio file.

[0074] It should be noted that details not disclosed in the multi-style speech generation apparatus 1000 of this embodiment can be found in the details disclosed in the multi-style speech generation method M10 of the above-described embodiments of this disclosure, and will not be repeated here.

[0075] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0076] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0077] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0078] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0079] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0080] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. This program can be stored in a readable storage medium, and when executed, it includes one or a combination of the steps of the method implementation. The storage medium can be volatile or non-volatile.

[0081] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0082] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the multi-style speech generation method of any of the above embodiments.

[0083] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0084] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method for generating multi-style speech according to any of the above embodiments.

[0085] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0087] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for generating multi-style speech, characterized by, The method comprises: obtaining style features of all audio samples in a training set through a style feature extraction module; clustering all obtained style features to obtain multiple style feature types; obtaining semantic features of the audio samples through a neural network vocoder, and obtaining a first loss value according to the semantic features and the style features; determining distances between the style features corresponding to the audio samples and the multiple style feature types, and calculating distance loss values corresponding to the distances according to a contrast loss function; updating parameters of a style transfer module according to a comprehensive loss value to make the distances smaller until a convergence condition is reached, to obtain a trained voice style transfer model, the comprehensive loss value comprising the first loss value and the distance loss values, the voice style transfer model comprising the neural network vocoder and the style transfer module, and the style transfer module comprising the style feature extraction module; and inputting a first audio into the neural network vocoder and inputting a second audio into the style feature extraction module to obtain a synthesized audio, a text content of the synthesized audio corresponding to the first audio and an audio style of the synthesized audio corresponding to the second audio. The method further comprises: inputting a spectrum of the audio sample into the neural network vocoder to obtain the semantic features; the inputting the spectrum of the audio sample into the neural network vocoder to obtain the semantic features comprises: performing one-dimensional convolution on the spectrum of the audio sample to obtain a first convolution result; and performing N times of anti-aliasing multi-period synthesis operations on the first convolution result to obtain the semantic features, each time the anti-aliasing multi-period synthesis operation comprising sequentially performing one-dimensional transpose convolution and anti-aliasing multi-period synthesis operation; and an output of the Nth anti-aliasing multi-period synthesis operation is the semantic features.

2. The generation method of claim 1, wherein, The method further comprises: performing matrix addition on the semantic features and the style features to obtain a to-be-processed feature; and performing nonlinear processing on the to-be-processed feature to obtain the first loss value.

3. The generation method of claim 2, wherein, The method further comprises: obtaining a spectrum of each audio sample in the training set, and extracting a style feature from the spectrum through the style feature extraction module.

4. The generation method of claim 1, wherein, Before clustering all obtained style features, the method further comprises: performing denoising on the style features extracted by the style feature extraction module through a multilayer perceptron. The method further comprises: for each style feature, determining a style feature type closest to the style feature from the multiple style feature types as a target type; and inputting distances between each style feature and the corresponding target type into a contrast loss function to obtain a distance loss value.

5. The generation method of claim 1, wherein, The style transfer module further comprises a personal feature learning module, and the method further comprises: inputting a unique identifier of the audio sample into the personal feature learning module for mapping, and inputting a mapping result and a corresponding style feature into an expectation minimization loss function to obtain an expectation loss value; Correspondingly, the first loss value is obtained according to the semantic feature, the style feature and the mapping result; Correspondingly, the comprehensive loss value further comprises the mapping result, and when the parameters of the style transfer module are updated according to the comprehensive loss value, the personal feature of the corresponding audio sample is learned by updating the parameters of the personal feature learning module to reduce the personal feature in the style feature, and the personal feature comprises an age feature and / or a gender feature and / or a timbre feature.

6. The generation method of claim 5, wherein, The text content of the first audio is the same as the text content of the audio sample in the training set, and when the first audio is input into the neural network vocoder, the unique identifier of the first audio is also input into the personal feature learning module to reduce the personal feature of the second audio existing in the obtained synthesized audio.

7. A multi-style speech generation apparatus characterized by comprising: Comprise: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the generation method of multi-style voice as claimed in any one of claims 1 to 6.

Images