Voice data processing method, device, electronic device and readable storage medium

Abstract

The present disclosure provides a method, device, electronic device and readable storage medium for processing speech data. The method includes: obtaining a vocal label and text data of a target speaker; inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates a target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; inputting the target vocal representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker. In this method, the timbre generation model can be used to classify and generate target vocal representation vectors of different types of voices and timbres; the speech synthesis model can be used to generate speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

Description

Technical Field

[0001] The present disclosure relates to the field of speech synthesis technology, and in particular to a method, device, electronic device and readable storage medium for processing speech data. Background Art

[0002] Speech is the most natural and commonly used form of communication between humans, and is also an important way for machines to interact with humans. In addition to the content of the language, speech also reflects semantically irrelevant information such as the speaker's timbre, emotions, and speaking style. Speech synthesis technology aims to synthesize corresponding speech based on input text information.

[0003] However, in the existing technology, typical speech synthesis technology can only synthesize the speech corresponding to the speaker in the training data, and cannot provide new speaker timbre outside the training data during the synthesis process, which cannot meet the needs of some application scenarios. In addition, the existing timbre generation scheme usually has a relatively simple speaker representation modeling, so it is difficult to control the characteristics of the timbre in this process, resulting in the problem of insufficient controllability of timbre generation. Summary of the invention

[0004] In view of this, the present disclosure provides a method, device, electronic device and readable storage medium for processing speech data, so as to at least partially solve the problem of insufficient controllability of timbre generation.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing speech data, the method comprising: obtaining a vocal label and text data of a target speaker; inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of sampling a multi-dimensional prior probability distribution and the vocal label; inputting the target vocal representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

[0006] In a second aspect, an embodiment of the present disclosure further provides a speech data processing device, the device comprising: a label and data acquisition module, used to acquire a vocal label and text data of a target speaker; a timbre generation model processing module, used to input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of sampling a multi-dimensional prior probability distribution and the vocal label; a speech synthesis model processing module, used to input the target vocal representation vector and text data into a pre-trained speech synthesis model to generate the speech data of the target speaker.

[0007] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the above-mentioned voice data processing method.

[0008] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the steps of the above-mentioned voice data processing method.

[0009] The disclosed embodiments provide a method, device, electronic device and readable storage medium for processing speech data, wherein a speech label is input into a pre-trained timbre generation model to generate a target speech representation vector of a target speaker; the target speech representation vector and text data are input into a pre-trained speech synthesis model to generate speech data of the target speaker. In this method, the timbre generation model can be used to classify and generate target speech representation vectors of different types of speech and timbre; the speech synthesis model can be used to generate speech data of the target speaker, thereby controlling the timbre characteristics of the speech data of the target speaker and improving the controllability of the speaker's timbre generation.

[0010] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0011] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a method for processing voice data provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of a timbre generation model and a speech synthesis model provided in an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram of a timbre generation model provided by an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a first sub-model provided in an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a second sub-model provided in an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of the structure of a voice data processing device provided by an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of the structure of another device for processing voice data provided by an embodiment of the present disclosure;

[0020] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0022] Speech is the most natural and commonly used form of communication between humans, and it is also an important way for machines and humans to interact. In addition to the content information of the language, speech also reflects semantically irrelevant information such as the speaker's timbre, emotion, and speaking style. Speech synthesis technology aims to synthesize corresponding speech based on input text information, but typical speech synthesis technology can only synthesize the speech corresponding to the speaker existing in the training data, and cannot provide unlimited speaker timbres in the synthesis process, and cannot meet the needs of some application scenarios. For example: dubbing for a large number of game characters, synthesizing speech for various personalized virtual humans, etc., which aims to generate speaker timbres that do not exist in the real world and provide speech synthesis services with unlimited speaker timbres.

[0023] In the prior art, a high-dimensional vector is usually used to represent the speaker's timbre, and the high-dimensional vector can be called a target voice representation vector. The target voice representation vector can be extracted by a pre-trained speaker timbre encoder or obtained by joint training in a speech synthesis model. In the timbre generation task, a target voice representation vector that does not exist in the training set is usually generated as the input of the speech synthesis model to generate the corresponding new speaker's speech.

[0024] Most existing methods use multi-dimensional probability distribution to model the target voice representation vector, and obtain a new target voice representation vector by sampling from the prior distribution during the inference process. For example, multi-dimensional Gaussian distribution is used to model speaker representation, that is, a multi-speaker speech synthesis model is first trained, the speaker embedding in the speech synthesis model is used as the speaker representation, and the region and gender are used as the conditions of the Gaussian distribution to model the speaker representation, and finally a new speaker representation is obtained by sampling from the Gaussian distribution.

[0025] However, the existing technology has the defects of insufficient controllability of the timbre generation process and a large demand for labeled speech synthesis training data. Since the existing timbre generation schemes usually use relatively simple prior distributions to model speaker representations, it is difficult to control the characteristics of the timbre in this process, resulting in insufficient controllability, such as controlling the age group and subjective feelings of the timbre (whether the timbre is thick or sweet). At the same time, since the entire model requires a large amount of speaker timbre data to generate a new timbre, a large amount of labeled speech synthesis data of speakers is required, and the demand for labeled speech synthesis training data is large, which brings a high data threshold.

[0026] Based on this, the embodiments of the present disclosure provide a method, device, electronic device and readable storage medium for processing voice data, which specifically provide a method for generating controllable timbre, which can improve the controllability of the timbre generation system and lower the data threshold.

[0027] To facilitate understanding of this embodiment, a method for processing voice data disclosed in an embodiment of the present disclosure is first introduced in detail.

[0028] This embodiment provides a method for processing voice data. Figure 1 A flowchart of a method for processing voice data is shown, and the method for processing voice data comprises the following steps:

[0029] Step S102, obtaining the voice tag and text data of the target speaker.

[0030] In this embodiment, different voice tags can be pre-set to characterize the characteristics of different speakers, which can include speaker characteristics or speaker voice characteristics. Exemplarily, the voice tags can include speaker age, speaker gender, and timbre. For example, the richness, sweetness, etc. can be set as voice tags.

[0031] For example, this embodiment can first set a reference audio for each timbre, such as the timbre of "deep", and select a voice that matches the deep timbre and a voice that does not match the deep timbre as the "1.0" and "0.0" reference values ​​of this characteristic respectively. Subsequent labeling of all people is performed based on this reference value. For example, if a person is between the two, "0.5" is labeled as the voice label.

[0032] The text data can be understood as the content that the target speaker wants to say, that is, the text content corresponding to the generated speech data of the target speaker. The text data can be obtained through the text processing module. For example, the target text of the target speaker can be obtained in this embodiment, and the text data of the target text can be obtained based on the pre-trained text processing module. The text data may include phoneme data. The text data in this embodiment may be phoneme-level data (e.g., k ei3 p u3 p ei2) obtained by processing the speech synthesis front end.

[0033] The input in this embodiment can be the vocal label and text data of the target speaker, and the output can be the speech data corresponding to the vocal label. In this embodiment, it is implemented through two parts, one part is the timbre generation model, which is used to generate the vocal representation vector according to the vocal label; the other part is the speech synthesis model, which is used to generate the speech data of the target speaker according to the vocal representation vector and text.

[0034] Step S104, inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of sampling a multi-dimensional prior probability distribution and the vocal label.

[0035] The input of the timbre generation model can be the sampling result of the vocal label and the multi-dimensional prior probability distribution, and the output can be the target vocal representation vector of the target speaker. The vocal representation vector can be a vector of fixed dimension that represents the timbre.

[0036] During the training process of the timbre generation model, the sample vocal representation vectors of various types of vocal samples with vocal annotations are transformed so that the transformed sample vocal representation vectors satisfy the multi-dimensional prior probability distribution. It can be understood that the timbre generation model has learned the transformation rule of transforming the vocal representation vector into a multi-dimensional prior probability distribution, and the transformation rule corresponds to various types of vocal annotations. Among them, the vocal annotation corresponds to the vocal label. The multi-dimensional prior probability distribution is a pre-specified multi-dimensional probability distribution, such as Gaussian distribution, Laplace distribution or Poisson distribution. In the use of the timbre generation model, the trained timbre generation model can obtain the target vocal representation vector by inverse transformation of the random sampling of the multi-dimensional prior probability distribution according to the vocal label and the inverse transformation of the learned transformation rule. Since the sampling of the multi-dimensional prior probability distribution is random, through the timbre generation model, it is theoretically possible to generate an unlimited number of new vocal representation vectors corresponding to the vocal label.

[0037] In this embodiment, the sampling result of sampling the multi-dimensional prior probability distribution may be a multi-dimensional sampling vector. The dimensions of the multi-dimensional sampling vector and the target human voice representation vector may be the same or different.

[0038] In some embodiments, the multidimensional sampling vector has the same dimension as the target vocal representation vector. If the dimension of the target vocal representation vector is higher than the multidimensional sampling vector, mapping the high dimension to the low dimension may reduce the accuracy of the mapping; if the dimension of the target vocal representation vector is lower than the multidimensional sampling vector, mapping the low dimension to the high dimension may increase the required computing resources.

[0039] In some embodiments, the vocal label includes a vocal classification label and a timbre feature label, and a first timbre generation model corresponding to the vocal classification label can be determined based on the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0040] Furthermore, the human voice classification label of this embodiment includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0041] Exemplarily, gender classification labels include male and female; age classification labels include childhood, youth, adulthood, and old age. According to different combinations of the above labels, the vocal classification labels are divided into eight categories. Each type of vocal classification label is used to train a timbre generation model separately, and the timbre feature labels are used as sample annotations in the process of training each timbre generation model. When using the timbre generation model, the first timbre generation model corresponding to the vocal classification label can be determined from the eight timbre generation models according to the vocal classification label, and then the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker. In this way, the timbre generation models of different vocal classifications can be made more accurate, thereby making the generated target vocal vector more accurate.

[0042] The timbre generation model can train one model for each type of voice label, or one model for each type of voice classification label. If only one timbre generation model is trained, it can be sampled on a 256-dimensional multi-dimensional prior probability distribution, and the speaker voice classification label and timbre feature label (male, young, deep) can be input together to obtain a predicted 256-dimensional target voice representation vector.

[0043] If a timbre generation model is trained for each type of vocal classification label, then a 256-dimensional multi-dimensional prior probability distribution can be sampled first to obtain a 256-dimensional vector, and this 256-dimensional vector is input into the first timbre generation model corresponding to the vocal classification label (assuming it is a model corresponding to male and young people), and the timbre feature label (thick or not thick) is input into the first timbre generation model to obtain a predicted 256-dimensional target vocal representation vector. The sampling process in the 256-dimensional multi-dimensional prior probability distribution can be 256 separate N (0, 1) sampling 256 numbers.

[0044] The timbre description label can represent the richness or sweetness of a voice. For example, the richness of a male voice or the sweetness of a female voice can be quantified as a timbre description label.

[0045] Furthermore, the timbre generation model in this embodiment can be a nonlinear reversible transformation flow model. Exemplarily, the above nonlinear reversible transformation flow model can be a Glow model, which can be based on a normalizing flow. The normalizing flow can be understood as a combination of a series of reversible functions, which can transform a simple distribution (such as a multidimensional prior probability distribution) into a complex distribution (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible.

[0046] Step S106: input the target voice representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

[0047] After the timbre generation model outputs the target human voice representation vector, the speech synthesis model cascaded with the timbre generation model can receive the target human voice representation vector output by the timbre generation model, and output the speech data of the target speaker according to the target human voice representation vector and the text data, and the output speech data can control both the timbre characteristics of the speaker and the speech content of the speaker. Among them, the speech synthesis model can be any trained model, which is used to output the input target human voice representation vector and text data as the speech data of the target speaker.

[0048] In some embodiments, text data can be input into a first sub-model of a pre-trained speech synthesis model, and the bottleneck features of the text data can be generated through the first sub-model; the target human voice representation vector and the bottleneck features can be input into a second sub-model of the speech synthesis model, and the speech data of the target speaker can be generated through the second sub-model.

[0049] See also Figure 2 A schematic diagram of a timbre generation model and a speech synthesis model is shown, wherein the timbre generation model can generate a controllable target voice representation vector according to a voice label; and the speech synthesis model can use text data and the target voice representation vector as input to synthesize corresponding speech data. The text data in this embodiment can be phoneme-level data obtained through speech synthesis front-end processing.

[0050] like Figure 2 As shown, the speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The input of the first sub-model can be text data, and the output can be a bottleneck feature; the bottleneck feature can be a feature that does not contain timbre-related information.

[0051] In some embodiments, the bottleneck feature includes a frame-level semantic feature, and the bottleneck feature may only include frame-level semantic-related information but not timbre-related information.

[0052] like Figure 2 As shown, the input of the second sub-model can be the bottleneck feature and the target vocal representation vector, and the output can be speech data. The target vocal representation vector can be output by the timbre generation model, so the target vocal representation vector contains timbre-related information, and the output speech data output by the second sub-model also contains timbre-related information.

[0053] After the speech synthesis model outputs the speech data of the target speaker, the speech data can be sent to a terminal device with communication function such as a mobile phone, a computer, a server, etc., and the above terminal device can play the speech data.

[0054] The disclosed embodiment provides a method for processing speech data, wherein a human voice label is input into a pre-trained timbre generation model to generate a target human voice representation vector of a target speaker; the target human voice representation vector and text data are input into a pre-trained speech synthesis model to generate speech data of the target speaker. In this method, the timbre generation model can be used to classify and generate target human voice representation vectors of different types of human voices and timbres; the speech synthesis model can be used to generate speech data of the target speaker, thereby controlling the timbre characteristics of the speech data of the target speaker and improving the controllability of the speaker's timbre generation.

[0055] The process of pre-training the timbre generation model in an optional embodiment may include: obtaining a first sample vocal characterization vector and a first sample annotation, and inputting the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; the timbre generation model transforms the first sample vocal characterization vector so that the transformed first sample vocal characterization vector obeys the multi-dimensional prior probability distribution; calculating the first loss function based on the transformed first sample vocal characterization vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model, and obtaining a trained timbre generation model corresponding to the first sample annotation.

[0056] In this embodiment, the speaker can be classified and labeled according to the voice classification label. Among them, the voice label corresponds to the voice label. In this embodiment, different voice labels can be pre-set to characterize the characteristics of different speakers. The characteristics may include the characteristics of the speaker or the voice characteristics of the speaker. Exemplarily, the voice label may include the speaker's age, the speaker's gender, and the timbre. For example, the richness, sweetness, etc. can be set as the voice label.

[0057] The timbre generation model in this embodiment can transform the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys the multi-dimensional prior probability distribution. The first loss function can be calculated based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. The first loss function can characterize the degree of similarity between the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. Optionally, the first loss function is used to characterize the difference between the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. The smaller the first loss function, the more similar the transformed first sample vocal representation vector is to the multi-dimensional prior probability distribution.

[0058] Therefore, this embodiment can take minimizing the first loss function as the training goal, and adjust the model parameters so that the transformed first sample vocal representation vector is most similar to the multidimensional prior probability distribution, thereby obtaining a trained timbre generation model corresponding to the first sample annotation. The multidimensional prior probability distribution is a pre-specified multidimensional probability distribution, such as Gaussian distribution, Laplace distribution or Poisson distribution.

[0059] In some embodiments, the vocal annotation of the first sample vocal representation vector may include a vocal classification annotation and a timbre feature annotation, wherein the vocal classification annotation may include a gender classification annotation and an age classification annotation, and the timbre feature annotation includes a preset timbre description annotation.

[0060] Accordingly, in some embodiments, different timbre generation models to be trained can be established in advance according to the vocal classification annotations; the target timbre generation model to be trained can be confirmed according to the vocal classification annotations; and the timbre feature annotations can be input into the target timbre generation model to be trained.

[0061] Exemplarily, the gender classification annotations include male and female; the age classification annotations include childhood, youth, adulthood, and old age. According to different combinations of the above annotations, the vocal classification annotations are divided into eight categories, and each type of vocal classification annotation is used to train a timbre generation model separately, and the timbre feature annotations are used as sample annotations in the process of training each timbre generation model. When training the timbre generation model, you can first confirm a timbre generation model as the target timbre generation model to be trained based on the vocal classification annotations, and then input the timbre feature annotations into the target timbre generation model to be trained to complete the training of the target timbre generation model to be trained. In this way, the timbre generation models of different vocal classifications can be made more accurate, so that the generated target vocal representation vectors are also more accurate.

[0062] In some embodiments, the multi-dimensional prior probability distribution in this embodiment may be a Gaussian distribution. High-speed distribution has the advantages of relatively simple distribution, common use, and convenient sampling.

[0063] In this embodiment, the Gaussian distribution of the target speaker can be determined based on the vocal annotation; the vocal annotation is input into a pre-trained timbre generation model, and a target vocal representation vector that satisfies the Gaussian distribution is output.

[0064] See also Figure 3 A schematic diagram of a timbre generation model is shown, wherein the timbre generation model may include a nonlinear reversible transform flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible one-dimensional convolution and a coupling layer.

[0065] The flow model in this embodiment can be a Glow model. The Glow model can be established on the basis of normalizing flow. Normalizing flow can be understood as a combination of a series of reversible functions, which can transform simple distribution (such as Gaussian distribution) into complex distribution (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible.

[0066] In some embodiments, the Glow model in this embodiment is a multi-layer stacked structure, each layer includes: a normalization layer (actnorm), an invertible one-dimensional convolution (invertible 1×1convolution) and a coupling layer (coupling layer). In this embodiment, the first sample annotation can be input into the affine coupling transformation layer, and the first sample vocal representation vector can be transformed through the affine coupling transformation layer.

[0067] In some embodiments, this embodiment may calculate the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, or Kullback-Leibler, KL divergence, and use the relative entropy as the first loss function.

[0068] like Figure 3 As shown in the figure, the Glow model can obtain the human voice label as a condition and transform the target human voice representation vector. The Glow model realizes the transformation between distributions. The relative entropy of the transformed first sample human voice representation vector and the Gaussian distribution is calculated as the first loss function to ensure that the transformed first sample human voice representation vector obeys the Gaussian distribution. In this way, during the inference process, a new target human voice representation vector can be obtained by sampling from the Gaussian distribution and undergoing the inverse transformation of the Glow model.

[0069] In some embodiments, the first sample human voice representation vector has the same dimension as the multi-dimensional prior probability distribution, so that the required computing resources can be reduced as much as possible without reducing the accuracy of the mapping.

[0070] In addition, when training the timbre generation model, the present embodiment can obtain a first sample human voice audio, and obtain a first human voice representation vector of the first sample human voice audio through the trained voiceprint feature model.

[0071] The timbre generation model and speech synthesis model in this embodiment can be trained separately. A pre-trained voiceprint model can be used to extract the first vocal representation vector of the first sample vocal audio, and the extracted target vocal representation vector can be used as the intermediate feature of the timbre generation model and the speech synthesis model for training the timbre generation model and the speech synthesis model.

[0072] The prior art requires a large amount of labeled data from multiple speakers for training. In order to reduce the demand for labeled training data, this embodiment can use a large amount of unlabeled data from multiple speakers and a small amount of labeled data from a single speaker to train the speech synthesis module. Figure 2 As shown, the speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model; the input of the first sub-model can be text data, and the output can be bottleneck features; the input of the second sub-model can be bottleneck features, and the output can be speech data.

[0073] In an optional embodiment, the speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model may include: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0074] The speech synthesis model in this embodiment includes a first sub-model and a second sub-model. The first sub-model and the second sub-model can be trained separately, and the training samples used by the first sub-model and the second sub-model can be the same or different.

[0075] See also Figure 4 The first sub-model can be trained using a small amount of labeled data from a single speaker to achieve the task of predicting frame-level bottleneck features from text data. Figure 5 The schematic diagram of a second sub-model shown in FIG. 1 is a diagram of a second sub-model, which can be trained with a large amount of unlabeled data of multiple speakers, which only has audio but no corresponding text annotations. Finally, the first sub-model and the second sub-model are cascaded to obtain the required speech synthesis model. In this way, the second sub-model of the speech synthesis model can be trained with unlabeled multi-speaker data, thereby reducing the demand for labeled training data.

[0076] When training the first sub-model and the second sub-model, the present embodiment can extract bottleneck features through a pre-trained speech recognition model. The speech recognition model can be an ASR (Automatic Speech Recognition) model, and the bottleneck features can be hidden features extracted using the pre-trained ASR model, for example: using the encoder of the Wenet speech recognition model to extract 512-dimensional bottleneck features, the sequence length of the extracted bottleneck features is consistent with the spectrum length.

[0077] In some embodiments, when training the first sub-model, a sample text can be generated based on the second sample human voice audio; the sample text is input into the first sub-model to obtain the first bottleneck feature; the second loss function is calculated based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; the first sub-model is trained with the goal of minimizing the second loss function to obtain a trained first sub-model.

[0078] When training the first sub-model, this embodiment can extract the first bottleneck feature of the annotated data of a single speaker (i.e., the second sample human voice audio) based on the pre-trained speech recognition model; and train the first sub-model of the speech synthesis model based on the first bottleneck feature and the bottleneck feature of the second sample human voice audio. When training the first sub-model, this embodiment can use the pre-trained speech recognition model to extract the bottleneck feature of the annotated data of a single speaker for training the first sub-model.

[0079] Among them, the second sample human voice audio can be a phoneme annotation, and the sample text is a phoneme text; this embodiment can convert the second sample human voice audio into text text, and obtain the phoneme text according to the text text.

[0080] In this embodiment, the first sub-model may be a Text2BN model, which may predict bottleneck features based on a given phoneme sequence, and the bottleneck features may include frame-level semantic features, which are related to semantics and have nothing to do with timbre. The second loss function of the Text2BN model may be an L2 loss function, which is calculated based on the predicted bottleneck features and the actual bottleneck features.

[0081] When training the first sub-model, you can obtain the annotated second sample human voice audio. Phoneme annotation is the result of manual correction after the text is converted into phonemes. The text processing module can be used to convert Chinese characters into phonemes. When it is finally used, the Chinese characters can be converted into phonemes first and then input into the Text2BN model as text.

[0082] Figure 4 The duration extension in can be understood as the expansion of the phoneme sequence into a frame sequence, and the frame sequence is several times longer than the phoneme sequence. Figure 4 The duration predictor in is used to predict how many frames each phoneme will be expanded into.

[0083] In some embodiments, when training the second sub-model, the present embodiment can obtain the second bottleneck feature and the second vocal characterization vector of the third sample vocal audio; input the second bottleneck feature and the second vocal characterization vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second vocal characterization vector, and the third sample vocal audio is used as the real sample of the generative adversarial network; the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

[0084] When training the second sub-model, the present embodiment can extract the second bottleneck feature of the unlabeled data of multiple speakers (i.e., the third sample human voice audio) based on the speech recognition model; extract the second human voice representation vector of the unlabeled data of multiple speakers based on the pre-trained voiceprint model; and train the second sub-model of the speech synthesis model based on the unlabeled data, the second bottleneck feature of the unlabeled data, and the second human voice representation vector of the unlabeled data. When training the second sub-model, the present embodiment can use the pre-trained speech recognition model to extract the bottleneck feature of the unlabeled data of multiple speakers for training the second sub-model.

[0085] In some embodiments, this embodiment can calculate a third loss function based on the generated Mel-spectrometer of the target speech and the Mel-spectrometer of the third sample vocal audio; and train the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0086] The second sub-model in this embodiment can be a BN2wav model, and the BN2wav model can be a generative adversarial network (GAN) structure. Therefore, the third loss function can include two, one is a mel spectrum loss function (mel loss), and the other is a discriminator loss function.

[0087] In some embodiments, the discriminator loss function may be: The above mel spectrum loss function can be: That is, the third loss function is calculated by comparing the Mel spectrum of the third sample vocal audio synthesized by the BN2wav model with the Mel spectrum of the target speech of the real speech.

[0088] In addition, the generator can generate speech data based on the bottleneck features of a large amount of speaker audio data, and the discriminator will judge the authenticity of the generated speech and the large amount of speaker audio data, so as to achieve the purpose of generating a real human voice based on the speech data. Moreover, in this process, the bottleneck feature can be connected to the target human voice representation vector, so that the target human voice representation vector can also be used as a generation factor in the generation process.

[0089] In summary, the first sub-model can be a Text2BN model, and the second sub-model can be a BN2wav model. The Text2BN model and the BN2wav model can be cascaded to obtain a complete speech synthesis model, which completes the function of synthesizing speech data according to text data and target vocal representation vectors. In the process of synthesizing speech data, the new target vocal representation vector generated by the timbre generation model can be input into the second sub-model as a synthesis condition. In some embodiments, the one-dimensional target vocal representation vector can be concatenated with the bottleneck feature after passing through a fully connected layer, and then input into the structure of the second sub-model.

[0090] The above method provided by the embodiment of the present disclosure can use a small amount of labeled data from a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data from multiple speakers to train the second sub-model of the speech synthesis model. There is no need to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0091] The above method provided by the embodiment of the present disclosure can use the vocal label that characterizes the timbre characteristics of the speaker as input, thereby controlling the timbre characteristics of the speaker's voice data and improving the controllability of the speaker's timbre generation. It is also possible to use a small amount of labeled data of a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data of multiple speakers to train the second sub-model of the speech synthesis model. It is not necessary to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0092] Corresponding to the above method embodiment, the present disclosure provides a device for processing speech data. Figure 6 A structural schematic diagram of a voice data processing device shown in FIG. 1 , the voice data processing device comprising:

[0093] A label and data acquisition module 61 is used to acquire the voice label and text data of the target speaker;

[0094] The timbre generation model processing module 62 is used to input the human voice label into the pre-trained timbre generation model to generate a target human voice representation vector of the target speaker, wherein the timbre generation model generates the target human voice representation vector according to the sampling result of sampling the multi-dimensional prior probability distribution and the human voice label;

[0095] The speech synthesis model processing module 63 is used to input the target human voice representation vector and text data into the pre-trained speech synthesis model to generate speech data of the target speaker.

[0096] The disclosed embodiment provides a speech data processing device, which inputs a human voice label into a pre-trained timbre generation model to generate a target human voice representation vector of a target speaker; and inputs the target human voice representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker. In this method, the timbre generation model can be used to classify and generate target human voice representation vectors of different types of human voices and timbres; and the speech synthesis model can be used to generate speech data of the target speaker, thereby controlling the timbre characteristics of the speech data of the target speaker and improving the controllability of the speaker's timbre generation.

[0097] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0098] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0099] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; a timbre generation model processing module is used to determine a first timbre generation model corresponding to the vocal classification label according to the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0100] In an optional embodiment of the present disclosure, the above-mentioned vocal classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0101] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model.

[0102] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model processing module is used to input text data into a first sub-model of a pre-trained speech synthesis model, and generate bottleneck features of the text data through the first sub-model; input the target human voice representation vector and the bottleneck features into a second sub-model of the speech synthesis model, and generate speech data of the target speaker through the second sub-model.

[0103] In an optional embodiment of the present disclosure, the above-mentioned bottleneck features include frame-level semantic features.

[0104] In an optional embodiment of the present disclosure, the above-mentioned label and data acquisition module is used to obtain the target text of the target speaker, and obtains the text data of the target text based on the pre-trained text processing module, and the text data includes phoneme data.

[0105] See also Figure 7 The structural schematic diagram of another speech data processing device shown in the figure, the speech data processing device also includes: a timbre generation model training module 64, connected to the timbre generation model processing module 62, the timbre generation model training module 64 is used to obtain a first sample vocal characterization vector and a first sample annotation, and input the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; the timbre generation model transforms the first sample vocal characterization vector so that the transformed first sample vocal characterization vector obeys the multi-dimensional prior probability distribution; calculates the first loss function based on the transformed first sample vocal characterization vector and the multi-dimensional prior probability distribution; takes minimizing the first loss function as the training goal, adjusts the model parameters of the timbre generation model, and obtains a trained timbre generation model corresponding to the first sample annotation.

[0106] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes vocal classification annotation and timbre feature annotation.

[0107] In an optional embodiment of the present disclosure, different timbre generation models to be trained are established in advance according to the vocal classification annotations; the above-mentioned timbre generation model training module is used to confirm the target timbre generation model to be trained according to the vocal classification annotations; and the timbre feature annotations are input into the target timbre generation model to be trained.

[0108] In an optional embodiment of the present disclosure, the multi-dimensional prior probability distribution is a Gaussian distribution.

[0109] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model training module is used to calculate the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, and use the relative entropy as the first loss function.

[0110] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupled transformation layer; the above-mentioned timbre generation model training module is used to input the first sample annotation into the affine coupled transformation layer.

[0111] In an optional embodiment of the present disclosure, the first sample vocal characterization vector and the multi-dimensional prior probability distribution have the same dimension.

[0112] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model training module is used to obtain a first sample vocal audio, and obtain a first vocal representation vector of the first sample vocal audio through a trained voiceprint feature model.

[0113] In an optional embodiment of the present disclosure, the speech synthesis model includes a first sub-model and a second sub-model, such as Figure 7 As shown, the speech data processing device also includes: a speech synthesis model training module 65, which is connected to the speech synthesis model processing module 63, and the speech synthesis model training module 65 is used to obtain a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; train the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; train the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0114] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is used to generate a sample text based on the second sample human voice audio; input the sample text into the first sub-model to obtain the first bottleneck feature; calculate the second loss function based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; train the first sub-model with the goal of minimizing the second loss function to obtain a trained first sub-model.

[0115] In an optional embodiment of the present disclosure, the second sample human voice audio is annotated with phonemes, and the sample text is a phoneme text; the speech synthesis model training module is used to convert the second sample human voice audio into text text, and obtain the phoneme text based on the text text.

[0116] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is used to obtain the second bottleneck feature and the second vocal characterization vector of the third sample vocal audio; the second bottleneck feature and the second vocal characterization vector are input into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second vocal characterization vector, and the third sample vocal audio is used as the real sample of the generative adversarial network; the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

[0117] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is also used to calculate the third loss function based on the generated Mel spectrum of the target speech and the Mel spectrum of the third sample vocal audio; and train the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0118] The voice data processing device provided in the embodiment of the present disclosure has the same technical features as the voice data processing method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0119] The present disclosure also provides an electronic device for executing the above-mentioned voice data processing method; see Figure 8 The structure diagram of an electronic device shown in FIG. 1 includes a memory 100 and a processor 101, wherein the memory 100 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 101 to perform the following steps:

[0120] Acquire a target speaker's vocal label and text data; input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; input the target vocal representation vector and text data into a pre-trained speech synthesis model to generate speech data for the target speaker.

[0121] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0122] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0123] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; the vocal label is input into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, including: determining a first timbre generation model corresponding to the vocal classification label according to the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0124] In an optional embodiment of the present disclosure, the above-mentioned vocal classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0125] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model.

[0126] In an optional embodiment of the present disclosure, the target voice representation vector and text data are input into a pre-trained speech synthesis model to generate the speech data of the target speaker, including: inputting the text data into a first sub-model of the pre-trained speech synthesis model, and generating the bottleneck features of the text data through the first sub-model; inputting the target voice representation vector and the bottleneck features into a second sub-model of the speech synthesis model, and generating the speech data of the target speaker through the second sub-model.

[0127] In an optional embodiment of the present disclosure, the above-mentioned bottleneck features include frame-level semantic features.

[0128] In an optional embodiment of the present disclosure, the above-mentioned obtaining text data of the target speaker includes: obtaining a target text of the target speaker, and obtaining text data of the target text based on a pre-trained text processing module, wherein the text data includes phoneme data.

[0129] In an optional embodiment of the present disclosure, the process of pre-training the timbre generation model includes: obtaining a first sample vocal characterization vector and a first sample annotation, and inputting the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; the timbre generation model transforms the first sample vocal characterization vector so that the transformed first sample vocal characterization vector obeys a multi-dimensional prior probability distribution; calculating a first loss function based on the transformed first sample vocal characterization vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model, and obtaining a trained timbre generation model corresponding to the first sample annotation.

[0130] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes vocal classification annotation and timbre feature annotation.

[0131] In an optional embodiment of the present disclosure, the above-mentioned pre-establishing different timbre generation models to be trained according to vocal classification annotations; inputting the first sample annotations into the timbre generation model includes: confirming the target timbre generation model to be trained according to the vocal classification annotations; and inputting the timbre feature annotations into the target timbre generation model to be trained.

[0132] In an optional embodiment of the present disclosure, the multi-dimensional prior probability distribution is a Gaussian distribution.

[0133] In an optional embodiment of the present disclosure, the above-mentioned calculation of the first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: calculating the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, and using the relative entropy as the first loss function.

[0134] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; the first sample annotation is input into the timbre generation model, including: inputting the first sample annotation into the affine coupling transformation layer.

[0135] In an optional embodiment of the present disclosure, the first sample vocal characterization vector and the multi-dimensional prior probability distribution have the same dimension.

[0136] In an optional embodiment of the present disclosure, the above-mentioned obtaining the first sample human voice characterization vector includes: obtaining a first sample human voice audio, and obtaining a first human voice characterization vector of the first sample human voice audio through a trained voiceprint feature model.

[0137] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0138] In an optional embodiment of the present disclosure, the above-mentioned training of the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio includes: generating a sample text according to the second sample human voice audio; inputting the sample text into the first sub-model to obtain the first bottleneck feature; calculating the second loss function according to the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; training the first sub-model with the goal of minimizing the second loss function to obtain the trained first sub-model.

[0139] In an optional embodiment of the present disclosure, the second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; generating the sample text according to the second sample human voice audio includes: converting the second sample human voice audio into text text, and obtaining the phoneme text according to the text text.

[0140] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample human voice audio includes: obtaining the second bottleneck feature and the second human voice characterization vector of the third sample human voice audio; inputting the second bottleneck feature and the second human voice characterization vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second human voice characterization vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

[0141] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample vocal audio also includes: calculating the third loss function based on the generated Mel-spectrogram of the target speech and the Mel-spectrogram of the third sample vocal audio; training the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0142] The disclosed embodiment inputs the human voice label into the pre-trained timbre generation model to generate the target human voice representation vector of the target speaker; inputs the target human voice representation vector and text data into the pre-trained speech synthesis model to generate the speech data of the target speaker. In this way, the timbre generation model can be used to classify and generate target human voice representation vectors of different types of human voices and timbres; the speech synthesis model can be used to generate the speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0143] Further, Figure 8 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0144] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0145] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0146] The embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned voice data processing method, and the following steps may be performed:

[0147] Acquire a target speaker's vocal label and text data; input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; input the target vocal representation vector and text data into a pre-trained speech synthesis model to generate speech data for the target speaker.

[0148] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0149] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0150] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; the vocal label is input into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, including: determining a first timbre generation model corresponding to the vocal classification label according to the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0151] In an optional embodiment of the present disclosure, the above-mentioned vocal classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0152] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model.

[0153] In an optional embodiment of the present disclosure, the target voice representation vector and text data are input into a pre-trained speech synthesis model to generate the speech data of the target speaker, including: inputting the text data into a first sub-model of the pre-trained speech synthesis model, and generating the bottleneck features of the text data through the first sub-model; inputting the target voice representation vector and the bottleneck features into a second sub-model of the speech synthesis model, and generating the speech data of the target speaker through the second sub-model.

[0154] In an optional embodiment of the present disclosure, the above-mentioned bottleneck features include frame-level semantic features.

[0155] In an optional embodiment of the present disclosure, the above-mentioned obtaining text data of the target speaker includes: obtaining a target text of the target speaker, and obtaining text data of the target text based on a pre-trained text processing module, wherein the text data includes phoneme data.

[0156] In an optional embodiment of the present disclosure, the process of pre-training the timbre generation model includes: obtaining a first sample vocal characterization vector and a first sample annotation, and inputting the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; the timbre generation model transforms the first sample vocal characterization vector so that the transformed first sample vocal characterization vector obeys a multi-dimensional prior probability distribution; calculating a first loss function based on the transformed first sample vocal characterization vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model, and obtaining a trained timbre generation model corresponding to the first sample annotation.

[0157] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes vocal classification annotation and timbre feature annotation.

[0158] In an optional embodiment of the present disclosure, the above-mentioned pre-establishing different timbre generation models to be trained according to vocal classification annotations; inputting the first sample annotations into the timbre generation model includes: confirming the target timbre generation model to be trained according to the vocal classification annotations; and inputting the timbre feature annotations into the target timbre generation model to be trained.

[0159] In an optional embodiment of the present disclosure, the multi-dimensional prior probability distribution is a Gaussian distribution.

[0160] In an optional embodiment of the present disclosure, the above-mentioned calculation of the first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: calculating the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, and using the relative entropy as the first loss function.

[0161] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; the first sample annotation is input into the timbre generation model, including: inputting the first sample annotation into the affine coupling transformation layer.

[0162] In an optional embodiment of the present disclosure, the first sample vocal characterization vector and the multi-dimensional prior probability distribution have the same dimension.

[0163] In an optional embodiment of the present disclosure, the above-mentioned obtaining the first sample human voice characterization vector includes: obtaining a first sample human voice audio, and obtaining a first human voice characterization vector of the first sample human voice audio through a trained voiceprint feature model.

[0164] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0165] In an optional embodiment of the present disclosure, the above-mentioned training of the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio includes: generating a sample text according to the second sample human voice audio; inputting the sample text into the first sub-model to obtain the first bottleneck feature; calculating the second loss function according to the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; training the first sub-model with the goal of minimizing the second loss function to obtain the trained first sub-model.

[0166] In an optional embodiment of the present disclosure, the second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; generating the sample text according to the second sample human voice audio includes: converting the second sample human voice audio into text text, and obtaining the phoneme text according to the text text.

[0167] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample human voice audio includes: obtaining the second bottleneck feature and the second human voice characterization vector of the third sample human voice audio; inputting the second bottleneck feature and the second human voice characterization vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second human voice characterization vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

[0168] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample vocal audio also includes: calculating the third loss function based on the generated Mel-spectrogram of the target speech and the Mel-spectrogram of the third sample vocal audio; training the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0169] The disclosed embodiment inputs the human voice label into the pre-trained timbre generation model to generate the target human voice representation vector of the target speaker; inputs the target human voice representation vector and text data into the pre-trained speech synthesis model to generate the speech data of the target speaker. In this way, the timbre generation model can be used to classify and generate target human voice representation vectors of different types of human voices and timbres; the speech synthesis model can be used to generate the speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0170] The voice data processing method, device, electronic device and computer program product of a readable storage medium provided in the embodiments of the present disclosure include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0172] In addition, in the description of the embodiments of the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0173] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0174] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0175] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for processing speech data, characterized in that: The method comprises: Get the target speaker's voice label and text data; The vocal tag is input into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer, and an affine coupling transformation layer; the timbre generation model obtains the target vocal representation vector by inverse transformation of a sampling result of sampling a multi-dimensional prior probability distribution according to the vocal tag and the inverse transformation of the learned transformation rule; The target human voice representation vector and the text data are input into a pre-trained speech synthesis model to generate speech data of the target speaker.

2. The method according to claim 1, characterized in that The sampling result of sampling the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

3. The method according to claim 2, characterized in that The multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

4. The method according to claim 1, characterized in that: The vocal tag includes a vocal classification tag and a timbre feature tag; the vocal tag is input into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, including: Determining a first timbre generation model corresponding to the vocal classification label according to the vocal classification label; the first timbre generation model is one of a plurality of pre-trained timbre generation models; The timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

5. The method according to claim 4, characterized in that The vocal classification labels include gender classification labels and age classification labels, and the timbre feature labels include preset timbre description labels.

6. The method according to claim 1, characterized in that Inputting the target human voice representation vector and the text data into a pre-trained speech synthesis model to generate speech data of the target speaker, including: Inputting the text data into a first sub-model of a pre-trained speech synthesis model, and generating a bottleneck feature of the text data through the first sub-model; The target human voice representation vector and the bottleneck feature are input into a second sub-model of the speech synthesis model, and the speech data of the target speaker is generated through the second sub-model.

7. The method according to claim 6, characterized in that The bottleneck features include frame-level semantic features.

8. The method according to claim 1, characterized in that Get the target speaker's text data, including: A target text of a target speaker is obtained, and text data of the target text is obtained based on a pre-trained text processing module, wherein the text data includes phoneme data.

9. The method according to claim 1, characterized in that: The process of pre-training the timbre generation model includes: Obtaining a first sample vocal characterization vector and a first sample annotation, and inputting the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; The timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys the multi-dimensional prior probability distribution; Calculate a first loss function according to the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; Taking minimization of the first loss function as a training objective, the model parameters of the timbre generation model are adjusted to obtain the timbre generation model that has been trained corresponding to the first sample annotation.

10. The method according to claim 9, characterized in that The vocal annotation of the first sample vocal representation vector includes a vocal classification annotation and a timbre feature annotation.

11. The method according to claim 10, characterized in that Preliminarily establishing different timbre generation models to be trained according to the vocal classification annotations; and inputting the first sample annotations into the timbre generation models, comprising: Confirming the target timbre generation model to be trained according to the vocal classification annotation; The timbre feature annotation is input into the target timbre generation model to be trained.

12. The method according to claim 9, characterized in that The multidimensional prior probability distribution is a Gaussian distribution.

13. The method according to claim 12, characterized in that Calculating a first loss function according to the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: The relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution is calculated, and the relative entropy is used as the first loss function.

14. The method according to claim 9, characterized in that Inputting the first sample annotation into the timbre generation model comprises: The first sample annotation is input into the affine coupled transformation layer.

15. The method according to claim 9, characterized in that The first sample vocal characterization vector has the same dimension as the multi-dimensional prior probability distribution.

16. The method according to claim 9, characterized in that Obtaining the first sample vocal representation vector, including: A first sample human voice audio is obtained, and a first human voice representation vector of the first sample human voice audio is obtained through a trained voiceprint feature model.

17. The method according to claim 1, characterized in that The speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: Obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; Training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; The second sub-model is trained by using the third sample vocal audio so that the trained second sub-model generates the target speech according to the bottleneck features of the vocal audio and the vocal representation vector.

18. The method according to claim 17, characterized in that Training the first sub-model by annotating the bottleneck features of the second sample human voice audio and the second sample human voice audio includes: Generate a sample text according to the second sample human voice audio; Inputting the sample text into the first sub-model to obtain a first bottleneck feature; Calculate a second loss function according to the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; The first sub-model is trained with the goal of minimizing the second loss function to obtain a trained first sub-model.

19. The method according to claim 18, characterized in that The second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; and generating the sample text according to the second sample human voice audio includes: The second sample human voice audio is converted into text, and the phoneme text is obtained according to the text.

20. The method according to claim 17, characterized in that Training the second sub-model by using the third sample human voice audio includes: Obtaining a second bottleneck feature and a second voice representation vector of the third sample voice audio; Inputting the second bottleneck feature and the second vocal representation vector into the second sub-model, wherein the second sub-model includes a generative adversarial network, wherein the generative adversarial network is used to generate a target speech according to the second bottleneck feature and the second vocal representation vector, and the third sample vocal audio is used as a real sample of the generative adversarial network; The second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

21. The method according to claim 20, characterized in that The second sub-model is trained by the third sample human voice audio, further comprising: Calculate a third loss function based on the generated Mel spectrum of the target speech and the Mel spectrum of the third sample human voice audio; The second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

22. A speech data processing device, characterized in that: The device comprises: A label and data acquisition module, used to obtain the target speaker's voice label and text data; A timbre generation model processing module is used to input the human voice label into a pre-trained timbre generation model to generate a target human voice representation vector of the target speaker, wherein the timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer, and an affine coupling transformation layer; the timbre generation model obtains the target human voice representation vector by inverse transformation of the sampling result of sampling the multi-dimensional prior probability distribution according to the human voice label and the inverse transformation of the learned transformation rule; The speech synthesis model processing module is used to input the target human voice representation vector and the text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

23. An electronic device, characterized in that: It includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the voice data processing method according to any one of claims 1 to 21.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the method for processing voice data according to any one of claims 1 to 21.

Images