Speech data processing method and apparatus, and electronic device and readable storage medium

By acquiring the voice tags and text data of the target speaker, and using the timbre generation model and speech synthesis model to generate the speech data of the target speaker, the problem of insufficient controllability of timbre generation and high data requirements in the existing technology is solved, thereby improving the controllability of timbre generation and reducing the data threshold.

WO2024239902A1PCT designated stage expired Publication Date: 2024-11-28NETEASE (HANGZHOU) NETWORK CO LTD

Patent Information

Application Number
PCT/CN2024/089809
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-04-25
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing speech synthesis technologies cannot generate new speaker timbres beyond the training data, have insufficient control over timbre generation, and require a large amount of labeled speech synthesis training data, resulting in a high data threshold.

Method used

By acquiring the voice tags and text data of the target speaker, and using a pre-trained timbre generation model and speech synthesis model, the target speaker's voice representation vector and speech data are generated. A multidimensional prior probability distribution is used for sampling, and a nonlinear reversible transform flow model is combined to improve the controllability of timbre generation. The speech synthesis model is trained with a small amount of labeled data and a large amount of unlabeled data.

Benefits of technology

This improves the controllability of the target speaker's timbre, reduces the need for labeled training data, and enhances the flexibility and efficiency of timbre generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089809_28112024_PF_FP_ABST
    Figure CN2024089809_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A speech data processing method, comprising: acquiring a human voice label and text data of a target speaker (S102); inputting the human voice label into a pre-trained timbre generation model, so as to generate a target human voice representation vector of the target speaker, wherein the timbre generation model generates the target human voice representation vector according to the human voice label and a sampling result that is obtained by means of performing sampling on a multi-dimensional prior probability distribution (S104); and inputting the target human voice representation vector and the text data into a pre-trained speech synthesis model, so as to generate speech data of the target speaker (S106). The target human voice representation vector having different types of human voice and timbres can be generated by means of classification by using the timbre generation model; and the speech data of the target speaker can be generated by using the speech synthesis model, such that timbre characteristics of the speech data of the target speaker are controlled, thereby improving the controllability of speaker timbre generation.
Need to check novelty before this filing date? Find Prior Art

Description

Voice data processing method, device, electronic device and readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202310595157.2, filed on May 24, 2023, entitled “Method, device, electronic device and readable storage medium for processing voice data”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of speech synthesis technology, and in particular to a method, device, electronic device, and readable storage medium for processing speech data. Background Art

[0004] Speech is the most natural and common form of communication between humans, and it is also a crucial method for interaction between machines and humans. Speech not only contains the linguistic content but also reflects semantically unrelated information such as the speaker's timbre, emotion, and speaking style. Speech synthesis technology aims to synthesize speech based on input text.

[0005] However, typical speech synthesis techniques can only synthesize speech corresponding to the speakers in the training data. They cannot provide the timbre of new speakers outside of the training data during the synthesis process, which makes them unsuitable for some application scenarios. Furthermore, existing timbre generation schemes often use relatively simple speaker representation modeling, making it difficult to control timbre characteristics during the process, resulting in insufficient controllability of timbre generation.

[0006] Summary of the Invention

[0007] According to one aspect of the present disclosure, a method for processing speech data is provided, the method comprising: obtaining a vocal label and text data of a target speaker; inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; and inputting the target vocal representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

[0008] According to one aspect of the present disclosure, a speech data processing device is also provided, the device including: a label and data acquisition module for acquiring a target speaker's vocal label and text data; a timbre generation model processing module for inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; and a speech synthesis model processing module for inputting the target vocal representation vector and text data into a pre-trained speech synthesis model to generate the target speaker's speech data.

[0009] According to one aspect of the present disclosure, an electronic device is also provided, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the steps of the above-mentioned voice data processing method.

[0010] According to one aspect of the present disclosure, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the steps of the above-mentioned voice data processing method.

[0011] The disclosed embodiments provide a method, apparatus, electronic device, and readable storage medium for processing speech data. These methods input a voice tag into a pre-trained timbre generation model to generate a target voice representation vector for a target speaker. The target voice representation vector and text data are then input into a pre-trained speech synthesis model to generate speech data for the target speaker. In this method, the timbre generation model can be used to generate target voice representation vectors for different types of voices and timbres. The speech synthesis model can then be used to generate speech data for the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 is a flowchart of a method for processing voice data provided by an embodiment of the present disclosure;

[0013] FIG2 is a schematic diagram of a timbre generation model and a speech synthesis model provided in an embodiment of the present disclosure;

[0014] FIG3 is a schematic diagram of a timbre generation model provided by an embodiment of the present disclosure;

[0015] FIG4 is a schematic diagram of a first sub-model provided in an embodiment of the present disclosure;

[0016] FIG5 is a schematic diagram of a second sub-model provided in an embodiment of the present disclosure;

[0017] FIG6 is a schematic structural diagram of a voice data processing device provided by an embodiment of the present disclosure;

[0018] FIG7 is a schematic structural diagram of another apparatus for processing voice data provided by an embodiment of the present disclosure;

[0019] FIG8 is a schematic structural diagram of one of the electronic devices provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0021] Speech is the most natural and common form of communication between humans, and it is also a crucial method for interaction between machines and humans. Speech not only contains the content of the language, but also reflects semantically unrelated information such as the speaker's timbre, emotion, and speaking style. Speech synthesis technology aims to synthesize speech based on input text. However, typical speech synthesis technology can only synthesize speech corresponding to speakers present in the training data, and cannot provide an unlimited number of speaker timbres during the synthesis process, thus failing to meet the needs of some application scenarios. For example, voice acting for a large number of game characters, synthesizing speech for various personalized virtual humans, and other scenarios aim to generate speaker timbres that do not exist in the real world and provide speech synthesis services with an unlimited number of speaker timbres.

[0022] Related technologies typically use a high-dimensional vector to represent a speaker's timbre. This high-dimensional vector can be called a target voice representation vector. This target voice representation vector can be extracted using a pre-trained speaker timbre encoder or trained jointly within a speech synthesis model. In timbre generation tasks, a target voice representation vector not present in the training set is typically generated as input to the speech synthesis model to generate the corresponding speech of a new speaker.

[0023] Most related methods use a multidimensional probability distribution to model the target voice representation vector, and then sample from the prior distribution during inference to obtain a new target voice representation vector. For example, a multidimensional Gaussian distribution is used to model speaker representations. This involves first training a multi-speaker speech synthesis model, using the speaker embeddings from that model as speaker representations, and then modeling the speaker representation using region and gender as conditions for the Gaussian distribution. Finally, a new speaker representation is obtained by sampling from the Gaussian distribution.

[0024] However, the related art has the drawbacks of insufficient controllability of the timbre generation process and a high demand for labeled speech synthesis training data. Since existing timbre generation schemes usually use relatively simple prior distributions to model speaker representations, it is difficult to control the characteristics of the timbre in this process, resulting in insufficient controllability, such as controlling the age group and subjective perception of the timbre (whether the timbre is rich or sweet). At the same time, since the entire model requires a large amount of speaker timbre data to generate new timbres, a large amount of labeled speech synthesis data of speakers is required. The demand for labeled speech synthesis training data is high, which brings a very high data threshold.

[0025] Based on this, the embodiments of the present disclosure provide a method, device, electronic device and readable storage medium for processing voice data, which specifically provides a method for generating controllable timbre, which can improve the controllability of the timbre generation system and lower the data threshold.

[0026] To facilitate understanding of this embodiment, a method for processing voice data disclosed in an embodiment of the present disclosure is first introduced in detail.

[0027] This embodiment provides a method for processing voice data. Referring to the flowchart of the method for processing voice data shown in FIG1 , the method for processing voice data includes the following steps:

[0028] Step S102: Acquire the target speaker's voice tag and text data.

[0029] In this embodiment, different voice tags can be pre-set to characterize different speakers. These tags can include speaker characteristics or voice characteristics. For example, the voice tags can include speaker age, gender, and timbre. For example, the richness or sweetness of a voice can be set as a voice tag.

[0030] For example, this embodiment can first set a reference audio for each timbre, such as the timbre of "deep", and select a voice that meets the deep timbre and a voice that does not meet the deep timbre as the "1.0" and "0.0" reference values ​​of this characteristic respectively. Subsequent labeling of all people is based on this reference value. For example, if a person is between the two, "0.5" is labeled as the voice label.

[0031] Text data can be understood as the content that the target speaker intends to say, that is, the textual content corresponding to the generated speech data of the target speaker. Text data can be obtained through a text processing module. For example, in this embodiment, the target text of the target speaker can be obtained, and the text data of the target text can be obtained based on a pre-trained text processing module. Text data can include phoneme data. In this embodiment, the text data can be phoneme-level data obtained through speech synthesis front-end processing (e.g., k ei3 p u3 p ei2).

[0032] In this embodiment, the input can be the target speaker's vocal label and text data, and the output can be the speech data corresponding to the vocal label. This is achieved through two parts: a timbre generation model, which generates a vocal representation vector based on the vocal label; and a speech synthesis model, which generates the target speaker's speech data based on the vocal representation vector and text.

[0033] In step S104, the vocal label is input into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label.

[0034] The input of the timbre generation model can be the sampling result of the human voice label and the multi-dimensional prior probability distribution, and the output can be the target human voice representation vector of the target speaker. The human voice representation vector can be a vector of fixed dimension that represents the timbre.

[0035] During the training process, the timbre generation model transforms the sample vocal representation vectors of various types of vocal samples with vocal annotations so that the transformed sample vocal representation vectors satisfy a multidimensional prior probability distribution. This can be understood as the timbre generation model learning the transformation rule for transforming the vocal representation vectors into a multidimensional prior probability distribution, and this transformation rule corresponds to each type of vocal annotation. The vocal annotation corresponds to the vocal label. The multidimensional prior probability distribution is a pre-specified multidimensional probability distribution, such as a Gaussian distribution, a Laplace distribution, or a Poisson distribution. During use, the trained timbre generation model can, based on the vocal labels and the inverse transformation of the learned transformation rule, inversely transform the random samples of the multidimensional prior probability distribution to obtain the target vocal representation vector. Since the sampling of the multidimensional prior probability distribution is random, the timbre generation model can theoretically generate an unlimited number of new vocal representation vectors corresponding to the vocal labels.

[0036] In this embodiment, the sampling result of sampling the multi-dimensional prior probability distribution may be a multi-dimensional sampling vector. The dimensions of the multi-dimensional sampling vector and the target voice representation vector may be the same or different.

[0037] In some embodiments, the multidimensional sampling vector has the same dimension as the target vocal representation vector. If the dimension of the target vocal representation vector is higher than that of the multidimensional sampling vector, mapping the high dimension to the low dimension may reduce the accuracy of the mapping; if the dimension of the target vocal representation vector is lower than that of the multidimensional sampling vector, mapping the low dimension to the high dimension may increase the required computing resources.

[0038] In some embodiments, the vocal label includes a vocal classification label and a timbre feature label, and a first timbre generation model corresponding to the vocal classification label can be determined based on the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0039] Furthermore, the voice classification labels of this embodiment include gender classification labels and age classification labels, and the timbre feature labels include preset timbre description labels.

[0040] Exemplarily, gender classification labels include male and female; age classification labels include childhood, youth, adulthood, and old age. Based on different combinations of the above labels, the vocal classification labels are divided into eight categories. A timbre generation model is trained separately for each type of vocal classification label, and the timbre feature labels are used as sample annotations in the training process of each timbre generation model. When using the timbre generation model, the first timbre generation model corresponding to the vocal classification label can be determined from the eight timbre generation models based on the vocal classification label, and the timbre feature labels can then be input into the first timbre generation model to generate a target vocal representation vector for the target speaker. In this way, the timbre generation models of different vocal classifications can be made more accurate, thereby making the generated target vocal vector more accurate.

[0041] The timbre generation model can be trained separately for each type of voice label, or separately for each type of voice classification label. If only one timbre generation model is trained, it can be sampled from a 256-dimensional multidimensional prior probability distribution, inputting both the speaker's voice classification label and the timbre feature label (male, young, full-bodied) to obtain a predicted 256-dimensional target voice representation vector.

[0042] If a timbre generative model is trained for each vocal classification label, the 256-dimensional multidimensional prior probability distribution can be sampled to obtain a 256-dimensional vector. This 256-dimensional vector is then input into the first timbre generative model corresponding to the vocal classification label (assuming it is the model corresponding to male and young people). The timbre feature label (full-bodied or not) is also input into the first timbre generative model to obtain a predicted 256-dimensional target vocal representation vector. The sampling process from the 256-dimensional multidimensional prior probability distribution can be 256 separate N(0, 1) samplings of 256 numbers.

[0043] The timbre description tag can represent the richness or sweetness of a voice. For example, the richness of a male voice or the sweetness of a female voice can be quantified as a timbre description tag.

[0044] Furthermore, the timbre generation model in this embodiment can be a nonlinear reversible transform flow model. Exemplarily, the nonlinear reversible transform flow model can be a Glow model. The Glow model can be based on a normalizing flow. The normalizing flow can be understood as a composite of a series of reversible functions that can transform a simple distribution (such as a multidimensional prior probability distribution) into a complex distribution (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible.

[0045] Step S106: Input the target voice representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

[0046] After the timbre generation model outputs the target vocal representation vector, a speech synthesis model cascaded with the timbre generation model can receive the target vocal representation vector output by the timbre generation model and output the target speaker's speech data based on the target vocal representation vector and text data. The output speech data can control both the speaker's timbre characteristics and the speaker's speech content. The speech synthesis model can be any trained model that is configured to output the target speaker's speech data based on the input target vocal representation vector and text data.

[0047] In some embodiments, text data can be input into the first sub-model of a pre-trained speech synthesis model, and the bottleneck features of the text data can be generated by the first sub-model; the target human voice representation vector and the bottleneck features can be input into the second sub-model of the speech synthesis model, and the speech data of the target speaker can be generated by the second sub-model.

[0048] Referring to Figure 2, a schematic diagram of a timbre generation model and a speech synthesis model is shown. The timbre generation model can generate a controllable target voice representation vector based on a voice tag; the speech synthesis model can take text data and the target voice representation vector as input to synthesize the corresponding speech data. The text data in this embodiment can be phoneme-level data processed by the speech synthesis front-end.

[0049] As shown in Figure 2, the speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The input of the first sub-model can be text data, and the output can be a bottleneck feature; the bottleneck feature can be a feature that does not contain timbre-related information.

[0050] In some embodiments, the bottleneck feature includes a frame-level semantic feature, which may only include frame-level semantic-related information but not timbre-related information.

[0051] As shown in Figure 2, the input of the second sub-model can be the bottleneck feature and the target vocal representation vector, and the output can be speech data. The target vocal representation vector can be output by the timbre generation model. Therefore, the target vocal representation vector contains timbre-related information, and the output speech data output by the second sub-model also contains timbre-related information.

[0052] After the speech synthesis model outputs the speech data of the target speaker, the speech data can be sent to a terminal device with communication functions such as a mobile phone, a computer, and a server, and the above terminal device can play the speech data.

[0053] The disclosed embodiments provide a method for processing speech data. This method involves inputting a vocal tag into a pre-trained timbre generation model to generate a target vocal representation vector for a target speaker. The target vocal representation vector and text data are then input into a pre-trained speech synthesis model to generate speech data for the target speaker. This method uses the timbre generation model to generate target vocal representation vectors for different types of voices and timbres. The speech synthesis model then generates speech data for the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0054] The process of pre-training the timbre generation model in an optional embodiment may include: obtaining a first sample vocal representation vector and a first sample annotation, and inputting the first sample vocal representation vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal representation vector; the timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys a multi-dimensional prior probability distribution; calculating a first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model to obtain a trained timbre generation model corresponding to the first sample annotation.

[0055] This embodiment can classify and label speakers according to voice classification labels. The voice labels correspond to the voice tags. In this embodiment, different voice labels can be pre-set to characterize the characteristics of different speakers. These characteristics can include speaker characteristics or voice characteristics. For example, the voice labels can include speaker age, gender, and timbre. For example, voice richness or sweetness can be set as voice labels.

[0056] The timbre generation model in this embodiment can transform the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys the multi-dimensional prior probability distribution. The first loss function can be calculated based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. The first loss function can characterize the degree of similarity between the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. Optionally, the first loss function is used to characterize the difference between the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution. The smaller the first loss function, the more similar the transformed first sample vocal representation vector is to the multi-dimensional prior probability distribution.

[0057] Therefore, this embodiment can use minimization of the first loss function as the training objective and adjust the model parameters to make the transformed first sample vocal representation vector most similar to the multidimensional prior probability distribution, thereby obtaining a trained timbre generation model corresponding to the first sample annotation. The multidimensional prior probability distribution is a pre-specified multidimensional probability distribution, such as a Gaussian distribution, a Laplace distribution, or a Poisson distribution.

[0058] In some embodiments, the vocal annotation of the first sample vocal representation vector may include a vocal classification annotation and a timbre feature annotation. The vocal classification annotation may include a gender classification annotation and an age classification annotation, and the timbre feature annotation may include a preset timbre description annotation.

[0059] Accordingly, in some embodiments, different timbre generation models to be trained can be established in advance according to the vocal classification annotations; the target timbre generation model to be trained can be confirmed according to the vocal classification annotations; and the timbre feature annotations can be input into the target timbre generation model to be trained.

[0060] Exemplarily, the gender classification annotations include male and female; the age classification annotations include childhood, youth, adulthood, and old age. According to the different combinations of the above annotations, the vocal classification annotations are divided into eight categories. Each category of vocal classification annotations is used to train a timbre generation model separately, and the timbre feature annotations are used as sample annotations in the process of training each timbre generation model. When training the timbre generation model, you can first confirm a timbre generation model as the target timbre generation model to be trained based on the vocal classification annotations, and then input the timbre feature annotations into the target timbre generation model to be trained to complete the training of the target timbre generation model to be trained. In this way, the timbre generation models of different vocal classifications can be made more accurate, thereby making the generated target vocal representation vector more accurate.

[0061] In some embodiments, the multi-dimensional prior probability distribution of this embodiment can be a Gaussian distribution. High-speed distribution has the advantages of relatively simple distribution, common use, and convenient sampling.

[0062] In this embodiment, the Gaussian distribution of the target speaker can be determined based on the vocal annotations; the vocal annotations are input into a pre-trained timbre generation model, and a target vocal representation vector that satisfies the Gaussian distribution is output.

[0063] Refer to the schematic diagram of a timbre generation model shown in Figure 3, wherein the timbre generation model may include a nonlinear reversible transformation flow model, which is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible one-dimensional convolution and a coupling layer.

[0064] The flow model in this embodiment can be a Glow model. The Glow model can be established on the basis of normalizing flow. Normalizing flow can be understood as a combination of a series of reversible functions, which can transform simple distributions (such as Gaussian distributions) into complex distributions (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible.

[0065] In some embodiments, the Glow model in this embodiment has a multi-layer stacked structure, each layer including: a normalization layer (actnorm), an invertible 1×1 convolution (invertible 1×1 convolution), and a coupling layer (coupling layer). In this embodiment, the first sample annotation can be input into the affine coupling transformation layer, and the first sample vocal representation vector can be transformed by the affine coupling transformation layer.

[0066] In some embodiments, this embodiment may calculate the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, or Kullback-Leibler, KL divergence, and use the relative entropy as the first loss function.

[0067] As shown in Figure 3, the Glow model can use the vocal label as a condition and transform the target vocal representation vector. The Glow model implements the transformation between distributions. The relative entropy of the transformed first-sample vocal representation vector and the Gaussian distribution is calculated as the first loss function to ensure that the transformed first-sample vocal representation vector follows the Gaussian distribution. In this way, during the inference process, a new target vocal representation vector can be obtained by sampling from the Gaussian distribution and then undergoing the inverse transformation of the Glow model.

[0068] In some embodiments, the first sample human voice representation vector and the multi-dimensional prior probability distribution have the same dimension, which can minimize the required computing resources without reducing the accuracy of the mapping.

[0069] In addition, when training the timbre generation model, this embodiment can obtain a first sample human voice audio, and obtain a first human voice representation vector of the first sample human voice audio through the trained voiceprint feature model.

[0070] The timbre generation model and speech synthesis model in this embodiment can be trained separately. A pre-trained voiceprint model can be used to extract the first vocal representation vector of the first sample vocal audio, and the extracted target vocal representation vector can be used as the intermediate feature of the timbre generation model and the speech synthesis model for training of the timbre generation model and the speech synthesis model.

[0071] Related techniques require a large amount of labeled data from multiple speakers for training. To reduce the need for labeled training data, this embodiment can use a large amount of unlabeled data from multiple speakers and a small amount of labeled data from a single speaker to train the speech synthesis module. To this end, as shown in Figure 2, the speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The input of the first sub-model can be text data, and the output can be bottleneck features; the input of the second sub-model can be bottleneck features, and the output can be speech data.

[0072] In an optional embodiment, the speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model may include: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature based on the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech based on the bottleneck feature of the human voice audio and the human voice representation vector.

[0073] The speech synthesis model in this embodiment includes a first sub-model and a second sub-model. The first sub-model and the second sub-model can be trained separately, and the training samples used by the first sub-model and the second sub-model can be the same or different.

[0074] Referring to Figure 4, a schematic diagram of a first sub-model is shown. The first sub-model can be trained using a small amount of labeled data from a single speaker to achieve the task of predicting frame-level bottleneck features from text data. Referring to Figure 5, a schematic diagram of a second sub-model is shown. The second sub-model can be trained using a large amount of unlabeled data from multiple speakers. This unlabeled data only has audio and no corresponding text annotations. Ultimately, the first sub-model and the second sub-model are cascaded to obtain the desired speech synthesis model. In this way, the second sub-model of the speech synthesis model can be trained using unlabeled multi-speaker data, thereby reducing the need for labeled training data.

[0075] When training the first sub-model and the second sub-model, this embodiment can extract bottleneck features using a pre-trained speech recognition model. The speech recognition model can be an ASR (Automatic Speech Recognition) model, and the bottleneck features can be hidden features extracted using the pre-trained ASR model. For example, a 512-dimensional bottleneck feature is extracted using an encoder of a Wenet speech recognition model, and the sequence length of the extracted bottleneck feature is consistent with the length of the spectrum.

[0076] In some embodiments, when training the first sub-model, a sample text can be generated based on the second sample human voice audio; the sample text is input into the first sub-model to obtain the first bottleneck feature; the second loss function is calculated based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; with the goal of minimizing the second loss function, the first sub-model is trained to obtain the trained first sub-model.

[0077] When training the first sub-model, this embodiment can extract the first bottleneck feature of the labeled data of a single speaker (i.e., the second sample human voice audio) based on a pre-trained speech recognition model; and train the first sub-model of the speech synthesis model based on the first bottleneck feature and the bottleneck feature of the second sample human voice audio. When training the first sub-model, this embodiment can use the pre-trained speech recognition model to extract the bottleneck feature of the labeled data of the single speaker for training the first sub-model.

[0078] The second sample human voice audio may be a phoneme annotation, and the sample text may be a phoneme text. In this embodiment, the second sample human voice audio may be converted into text, and the phoneme text may be obtained based on the text.

[0079] In this embodiment, the first sub-model can be a Text2BN model. The Text2BN model can predict bottleneck features based on a given phoneme sequence. The bottleneck features can include frame-level semantic features that are related to semantics and not to timbre. The second loss function of the Text2BN model can be an L2 loss function, which is calculated based on the predicted bottleneck features and the actual bottleneck features.

[0080] When training the first sub-model, you can obtain a second sample of annotated human voice audio. Phoneme annotation is the result of manual correction after the text is converted into phonemes. The text processing module can convert Chinese characters into phonemes. When finally used, you can first convert Chinese characters into phonemes and then input them as text into the Text2BN model.

[0081] The duration expansion in Figure 4 can be understood as the expansion of a phoneme sequence into a frame sequence, where the frame sequence is several times longer than the phoneme sequence. The duration predictor in Figure 4 is used to predict how many frames each phoneme will be expanded into.

[0082] In some embodiments, when training the second sub-model, this embodiment can obtain the second bottleneck feature and the second voice representation vector of the third sample human voice audio; input the second bottleneck feature and the second human voice representation vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second human voice representation vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; with the goal of minimizing the discriminator loss function of the generative adversarial network, the second sub-model is trained to obtain a trained second sub-model.

[0083] When training the second sub-model, this embodiment can extract the second bottleneck feature of the multi-speaker unlabeled data (i.e., the third sample human voice audio) based on the speech recognition model; extract the second human voice representation vector of the multi-speaker unlabeled data based on the pre-trained voiceprint model; and train the second sub-model of the speech synthesis model based on the unlabeled data, the second bottleneck feature of the unlabeled data, and the second human voice representation vector of the unlabeled data. When training the second sub-model, this embodiment can use the pre-trained speech recognition model to extract the bottleneck feature of the multi-speaker unlabeled data for training the second sub-model.

[0084] In some embodiments, this embodiment can calculate a third loss function based on the generated mel spectrum of the target speech and the mel spectrum of the third sample human voice audio; and train the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0085] The second sub-model in this embodiment may be a BN2wav model, which may be a Generative Adversarial Networks (GAN) structure. Therefore, the third loss function may include two functions: a mel spectrum loss function (mel loss) and a discriminator loss function.

[0086] In some embodiments, the discriminator loss function may be: The above mel spectrum loss function can be: That is, the third loss function is calculated by comparing the Mel spectrum of the third sample human voice audio synthesized by the BN2wav model with the Mel spectrum of the target speech of the real speech.

[0087] Furthermore, the generator can generate speech data based on the bottleneck features of a large amount of speaker audio data. The discriminator then determines the authenticity of the generated speech compared to the large amount of speaker audio data, thereby achieving the goal of generating realistic human voices from the speech data. Furthermore, during this process, the bottleneck features can be connected to the target human voice representation vector, allowing the target human voice representation vector to be used as a generation factor during the generation process.

[0088] In summary, the first sub-model can be a Text2BN model, and the second sub-model can be a BN2wav model. The Text2BN model and the BN2wav model can be cascaded to form a complete speech synthesis model, which can synthesize speech data based on text data and a target voice representation vector. During the speech data synthesis process, the new target voice representation vector generated by the timbre generation model can be input into the second sub-model as a synthesis condition. In some embodiments, the one-dimensional target voice representation vector can be passed through a fully connected layer, concatenated with the bottleneck feature, and then input into the structure of the second sub-model.

[0089] The above method provided by the embodiment of the present disclosure can use a small amount of labeled data from a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data from multiple speakers to train the second sub-model of the speech synthesis model. There is no need to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0090] The method provided by the embodiments of the present disclosure can use vocal labels representing the timbre characteristics of a speaker as input, thereby controlling the timbre characteristics of the speaker's speech data and improving the controllability of speaker timbre generation. Furthermore, a small amount of labeled data from a single speaker can be used to train the first sub-model of the speech synthesis model, while a large amount of unlabeled data from multiple speakers can be used to train the second sub-model of the speech synthesis model. This eliminates the need to use only labeled data to train the speech synthesis model, thus reducing the demand for labeled training data.

[0091] Corresponding to the above method embodiment, the present disclosure provides a device for processing voice data. As shown in FIG6 , a schematic diagram of the structure of a voice data processing device is shown. The voice data processing device includes:

[0092] The label and data acquisition module 61 is used to obtain the target speaker's voice label and text data;

[0093] The timbre generation model processing module 62 is configured to input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label;

[0094] The speech synthesis model processing module 63 is used to input the target human voice representation vector and text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

[0095] The disclosed embodiments provide a speech data processing device that inputs a vocal tag into a pre-trained timbre generation model to generate a target vocal representation vector for a target speaker. The target vocal representation vector and text data are then input into a pre-trained speech synthesis model to generate speech data for the target speaker. In this method, the timbre generation model can be used to generate target vocal representation vectors for different types of voices and timbres. The speech synthesis model can then be used to generate speech data for the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0096] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0097] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0098] If the target vocal representation vector has a higher dimension than the multidimensional sampling vector, mapping it to a lower dimension may reduce mapping accuracy. If the target vocal representation vector has a lower dimension than the multidimensional sampling vector, mapping it to a higher dimension may increase the required computing resources. By making the multidimensional sampling vector and the target vocal representation vector have the same dimension, we can minimize computing resource consumption while ensuring mapping accuracy.

[0099] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; the timbre generation model processing module is used to determine a first timbre generation model corresponding to the vocal classification label based on the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; the timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

[0100] The timbre generation model processing module in this embodiment determines the first timbre generation model through the voice classification label, and uses the timbre feature label as input to the first timbre generation model to generate a target voice representation vector for the target speaker. That is, the timbre generation model can be used to classify and generate target voice representation vectors for different types of voices and timbres. This can make the timbre generation models of different voice classifications more accurate, thereby making the generated target voice vectors more accurate.

[0101] In an optional embodiment of the present disclosure, the above-mentioned voice classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0102] In this embodiment, the timbre generation models of different vocal categories can be made more accurate, thereby making the generated target vocal vector more accurate.

[0103] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model. In this embodiment, the timbre generation model is a nonlinear reversible transform flow model, which can be established on the basis of normalizing flow, and can transform simple distributions (such as multidimensional prior probability distributions) into complex distributions (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible. This further improves the accuracy of the timbre generation model, thereby making the generated target vocal vector more accurate.

[0104] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model processing module is used to input text data into the first sub-model of a pre-trained speech synthesis model, and generate bottleneck features of the text data through the first sub-model; input the target human voice representation vector and bottleneck features into the second sub-model of the speech synthesis model, and generate speech data of the target speaker through the second sub-model.

[0105] The speech synthesis model processing module in this embodiment can classify and generate target voice representation vectors of different types of human voices and timbres through the timbre generation model, so that the speech synthesis model can subsequently generate the speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0106] In an optional embodiment of the present disclosure, the bottleneck features include frame-level semantic features.

[0107] In an optional embodiment of the present disclosure, the above-mentioned label and data acquisition module is used to obtain the target text of the target speaker, and obtains text data of the target text based on a pre-trained text processing module, and the text data includes phoneme data.

[0108] The label and data acquisition module in this embodiment, the text data in this embodiment may be phoneme-level data obtained through speech synthesis front-end processing, which can improve the efficiency and accuracy of acquiring the target text of the target speaker.

[0109] Referring to the structural schematic diagram of another speech data processing device shown in Figure 7, the speech data processing device also includes: a timbre generation model training module 64, which is connected to the timbre generation model processing module 62, and the timbre generation model training module 64 is used to obtain a first sample vocal representation vector and a first sample annotation, and input the first sample vocal representation vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal representation vector; the timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys the multi-dimensional prior probability distribution; calculates the first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; takes minimizing the first loss function as the training goal, adjusts the model parameters of the timbre generation model, and obtains a trained timbre generation model corresponding to the first sample annotation.

[0110] In this embodiment, when training a timbre generation model using the above module, a timbre generation model can be first identified as a target timbre generation model to be trained based on the vocal classification annotations. The timbre feature annotations are then input into the target timbre generation model to complete the training of the target timbre generation model. This makes the timbre generation models for different vocal classifications more accurate, thereby also making the generated target vocal representation vectors more accurate.

[0111] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes a vocal classification annotation and a timbre feature annotation.

[0112] In an optional embodiment of the present disclosure, different timbre generation models to be trained are established in advance according to the vocal classification annotations; the above-mentioned timbre generation model training module is used to confirm the target timbre generation model to be trained according to the vocal classification annotations; and the timbre feature annotations are input into the target timbre generation model to be trained.

[0113] In this embodiment, a timbre generation model can be first identified based on the vocal classification annotations as the target timbre generation model to be trained, and then the timbre feature annotations can be input into the target timbre generation model to complete the training of the target timbre generation model to be trained. In this way, the timbre generation models for different vocal classifications can be made more accurate, thereby making the generated target vocal representation vector more accurate.

[0114] In an optional embodiment of the present disclosure, the multidimensional prior probability distribution is a Gaussian distribution, which has advantages such as relatively simple distribution, common use, and convenient sampling.

[0115] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model training module is used to calculate the relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution, and use the relative entropy as the first loss function.

[0116] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; the above-mentioned timbre generation model training module is used to input the first sample annotation into the affine coupling transformation layer.

[0117] In an optional embodiment of the present disclosure, the first sample human voice representation vector and the multi-dimensional prior probability distribution have the same dimension. This embodiment can minimize the required computing resources without reducing the accuracy of the mapping.

[0118] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model training module is used to obtain a first sample human voice audio, and obtain a first human voice representation vector of the first sample human voice audio through the trained voiceprint feature model.

[0119] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model includes a first sub-model and a second sub-model, as shown in Figure 7. The speech data processing device also includes: a speech synthesis model training module 65, which is connected to the speech synthesis model processing module 63, and the speech synthesis model training module 65 is used to obtain the second sample human voice audio, the bottleneck feature annotation of the second sample human voice audio, and the third sample human voice audio; train the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; train the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0120] The speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The first sub-model can be trained using a small amount of labeled data from a single speaker to predict frame-level bottleneck features from text data. The second sub-model can be trained using a large amount of unlabeled data from multiple speakers. This unlabeled data only contains audio and no corresponding text annotations. Ultimately, the first and second sub-models are cascaded to obtain the desired speech synthesis model. In this way, the second sub-model of the speech synthesis model can be trained using unlabeled multi-speaker data, thereby reducing the need for labeled training data.

[0121] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is used to generate a sample text based on the second sample human voice audio; input the sample text into the first sub-model to obtain the first bottleneck feature; calculate the second loss function based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; train the first sub-model with the goal of minimizing the second loss function to obtain a trained first sub-model.

[0122] In an optional embodiment of the present disclosure, the second sample human voice audio is phoneme-tagged, and the sample text is phoneme text; the speech synthesis model training module is used to convert the second sample human voice audio into text and obtain the phoneme text based on the text.

[0123] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is used to obtain the second bottleneck feature and the second voice representation vector of the third sample human voice audio; the second bottleneck feature and the second voice representation vector are input into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second voice representation vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

[0124] The above method provided by the embodiment of the present disclosure can use a small amount of labeled data from a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data from multiple speakers to train the second sub-model of the speech synthesis model. There is no need to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0125] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model training module is also used to calculate a third loss function based on the generated mel spectrum of the target speech and the mel spectrum of the third sample human voice audio; and the second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0126] The method provided by the embodiments of the present disclosure can use vocal labels representing the timbre characteristics of a speaker as input, thereby controlling the timbre characteristics of the speaker's speech data and improving the controllability of speaker timbre generation. Furthermore, a small amount of labeled data from a single speaker can be used to train the first sub-model of the speech synthesis model, while a large amount of unlabeled data from multiple speakers can be used to train the second sub-model of the speech synthesis model. This eliminates the need to use only labeled data to train the speech synthesis model, thus reducing the demand for labeled training data.

[0127] The voice data processing device provided in the embodiment of the present disclosure has the same technical features as the voice data processing method provided in the above embodiment, and can therefore solve the same technical problems and achieve the same technical effects.

[0128] The present disclosure also provides an electronic device for executing the above-mentioned voice data processing method. Referring to FIG8 , a schematic diagram of the structure of an electronic device is shown. The electronic device includes a memory 100 and a processor 101. The memory 100 is used to store one or more computer instructions. The one or more computer instructions are executed by the processor 101 to perform the following steps:

[0129] Acquire a target speaker's vocal label and text data; input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; input the target vocal representation vector and text data into a pre-trained speech synthesis model to generate the target speaker's speech data.

[0130] In this disclosed embodiment, a vocal tag is input into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker. The target vocal representation vector and text data are then input into a pre-trained speech synthesis model to generate speech data for the target speaker. This approach allows the timbre generation model to generate target vocal representation vectors for different types of voices and timbres. The speech synthesis model can then generate speech data for the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0131] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0132] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0133] In this embodiment, if the target vocal representation vector has a higher dimension than the multidimensional sampling vector, mapping the higher dimension to the lower dimension may reduce mapping accuracy. If the target vocal representation vector has a lower dimension than the multidimensional sampling vector, mapping the lower dimension to the higher dimension may increase the required computing resources. By ensuring that the multidimensional sampling vector and the target vocal representation vector have the same dimension, computing resource consumption can be minimized while ensuring mapping accuracy.

[0134] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker includes: determining a first timbre generation model corresponding to the vocal classification label based on the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; and inputting the timbre feature label into the first timbre generation model to generate a target vocal representation vector for the target speaker.

[0135] In this embodiment, the first timbre generation model is determined by the voice classification label, and the timbre feature label is used as input to the first timbre generation model to generate a target voice representation vector of the target speaker. That is, the timbre generation model can be used to classify and generate target voice representation vectors of different types of voices and timbres. This can make the timbre generation models of different voice classifications more accurate, thereby making the generated target voice vectors more accurate.

[0136] In an optional embodiment of the present disclosure, the above-mentioned voice classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0137] In this embodiment, the timbre generation models of different vocal categories can be made more accurate, thereby making the generated target vocal vector more accurate.

[0138] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model.

[0139] In this embodiment, the timbre generation model is determined to be a nonlinear reversible transform flow model, which can transform a simple distribution (such as a multidimensional prior probability distribution) into a complex distribution (such as the target vocal representation vector in this embodiment), and the transformation is completely reversible, further improving the accuracy of the timbre generation model, thereby making the generated target vocal vector more accurate.

[0140] In an optional embodiment of the present disclosure, the above-mentioned inputting the target human voice representation vector and text data into a pre-trained speech synthesis model to generate the speech data of the target speaker includes: inputting the text data into a first sub-model of the pre-trained speech synthesis model, and generating the bottleneck features of the text data through the first sub-model; inputting the target human voice representation vector and the bottleneck features into a second sub-model of the speech synthesis model, and generating the speech data of the target speaker through the second sub-model.

[0141] The speech synthesis model processing module in this embodiment can classify and generate target voice representation vectors of different types of human voices and timbres through the timbre generation model, so that the speech synthesis model can subsequently generate the speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0142] In an optional embodiment of the present disclosure, the bottleneck features include frame-level semantic features.

[0143] In an optional embodiment of the present disclosure, the above-mentioned obtaining text data of the target speaker includes: obtaining a target text of the target speaker, and obtaining text data of the target text based on a pre-trained text processing module, wherein the text data includes phoneme data.

[0144] The label and data acquisition module in this embodiment, the text data in this embodiment may be phoneme-level data obtained through speech synthesis front-end processing, which can improve the efficiency and accuracy of acquiring the target text of the target speaker.

[0145] In an optional embodiment of the present disclosure, the process of pre-training the timbre generation model includes: obtaining a first sample vocal representation vector and a first sample annotation, and inputting the first sample vocal representation vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal representation vector; the timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys a multi-dimensional prior probability distribution; calculating a first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model to obtain a trained timbre generation model corresponding to the first sample annotation.

[0146] In this embodiment, when training a timbre generation model using the above module, a timbre generation model can be first identified as a target timbre generation model to be trained based on the vocal classification annotations. The timbre feature annotations are then input into the target timbre generation model to complete the training of the target timbre generation model. This makes the timbre generation models for different vocal classifications more accurate, thereby also making the generated target vocal representation vectors more accurate.

[0147] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes a vocal classification annotation and a timbre feature annotation.

[0148] In an optional embodiment of the present disclosure, the above-mentioned different timbre generation models to be trained are established in advance according to the human voice classification annotations; the first sample annotation is input into the timbre generation model, including: confirming the target timbre generation model to be trained according to the human voice classification annotations; and inputting the timbre feature annotations into the target timbre generation model to be trained.

[0149] In this embodiment, a timbre generation model can be first identified based on the vocal classification annotations as the target timbre generation model to be trained, and then the timbre feature annotations can be input into the target timbre generation model to complete the training of the target timbre generation model to be trained. In this way, the timbre generation models for different vocal classifications can be made more accurate, thereby making the generated target vocal representation vector more accurate.

[0150] In an optional embodiment of the present disclosure, the multidimensional prior probability distribution is a Gaussian distribution.

[0151] The multi-dimensional prior probability distribution in this embodiment can be a Gaussian distribution. High-speed distribution has the advantages of relatively simple distribution, common use, and convenient sampling.

[0152] In an optional embodiment of the present disclosure, the above-mentioned calculation of the first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: calculating the relative entropy of the transformed first sample vocal representation vector and the Gaussian distribution, and using the relative entropy as the first loss function.

[0153] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; the first sample annotation is input into the timbre generation model, including: inputting the first sample annotation into the affine coupling transformation layer.

[0154] In an optional embodiment of the present disclosure, the first sample human voice representation vector and the multi-dimensional prior probability distribution have the same dimension. This embodiment can minimize the required computing resources without reducing the accuracy of the mapping.

[0155] In an optional embodiment of the present disclosure, the above-mentioned obtaining of the first sample human voice representation vector includes: obtaining a first sample human voice audio, and obtaining a first human voice representation vector of the first sample human voice audio through a trained voiceprint feature model.

[0156] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0157] The speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The first sub-model can be trained using a small amount of labeled data from a single speaker to predict frame-level bottleneck features from text data. The second sub-model can be trained using a large amount of unlabeled data from multiple speakers. This unlabeled data only contains audio and no corresponding text annotations. Ultimately, the first and second sub-models are cascaded to obtain the desired speech synthesis model. In this way, the second sub-model of the speech synthesis model can be trained using unlabeled multi-speaker data, thereby reducing the need for labeled training data.

[0158] In an optional embodiment of the present disclosure, the above-mentioned training of the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio includes: generating a sample text based on the second sample human voice audio; inputting the sample text into the first sub-model to obtain the first bottleneck feature; calculating the second loss function based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; training the first sub-model with the goal of minimizing the second loss function to obtain the trained first sub-model.

[0159] In an optional embodiment of the present disclosure, the second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; generating the sample text based on the second sample human voice audio includes: converting the second sample human voice audio into text text, and obtaining the phoneme text based on the text text.

[0160] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample human voice audio includes: obtaining the second bottleneck feature and the second human voice representation vector of the third sample human voice audio; inputting the second bottleneck feature and the second human voice representation vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second human voice representation vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; with the goal of minimizing the discriminator loss function of the generative adversarial network, the second sub-model is trained to obtain the trained second sub-model.

[0161] The above method provided by the embodiment of the present disclosure can use a small amount of labeled data from a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data from multiple speakers to train the second sub-model of the speech synthesis model. There is no need to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0162] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model by the third sample human voice audio also includes: calculating the third loss function based on the generated mel spectrum of the target speech and the mel spectrum of the third sample human voice audio; training the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0163] The method provided by the embodiments of the present disclosure can use vocal labels representing the timbre characteristics of a speaker as input, thereby controlling the timbre characteristics of the speaker's speech data and improving the controllability of speaker timbre generation. Furthermore, a small amount of labeled data from a single speaker can be used to train the first sub-model of the speech synthesis model, while a large amount of unlabeled data from multiple speakers can be used to train the second sub-model of the speech synthesis model. This eliminates the need to use only labeled data to train the speech synthesis model, thus reducing the demand for labeled training data.

[0164] Furthermore, the electronic device shown in FIG8 further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0165] Memory 100 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between the system network element and at least one other network element is achieved through at least one communication interface 103 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, etc. Bus 102 may be an ISA bus, a PCI bus, or an EISA bus. Buses can be classified as address buses, data buses, control buses, etc. For ease of illustration, FIG8 uses only one bidirectional arrow, but this does not mean that there is only one bus or only one type of bus.

[0166] The processor 101 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 101 or by instructions in the form of software. The above-mentioned processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and, in conjunction with its hardware, completes the steps of the method of the aforementioned embodiment.

[0167] The present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned method for processing voice data, and the following steps may be performed:

[0168] Acquire a target speaker's vocal label and text data; input the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of a multi-dimensional prior probability distribution and the vocal label; input the target vocal representation vector and text data into a pre-trained speech synthesis model to generate the target speaker's speech data.

[0169] In this disclosed embodiment, a vocal tag is input into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker. The target vocal representation vector and text data are then input into a pre-trained speech synthesis model to generate speech data for the target speaker. This approach allows the timbre generation model to generate target vocal representation vectors for different types of voices and timbres. The speech synthesis model can then generate speech data for the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0170] In an optional embodiment of the present disclosure, the sampling result of the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

[0171] In an optional embodiment of the present disclosure, the multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

[0172] If the target vocal representation vector has a higher dimension than the multidimensional sampling vector, mapping it to a lower dimension may reduce mapping accuracy. If the target vocal representation vector has a lower dimension than the multidimensional sampling vector, mapping it to a higher dimension may increase the required computing resources. By making the multidimensional sampling vector and the target vocal representation vector have the same dimension, we can minimize computing resource consumption while ensuring mapping accuracy.

[0173] In an optional embodiment of the present disclosure, the above-mentioned vocal label includes a vocal classification label and a timbre feature label; inputting the vocal label into a pre-trained timbre generation model to generate a target vocal representation vector for the target speaker includes: determining a first timbre generation model corresponding to the vocal classification label based on the vocal classification label; the first timbre generation model is one of multiple pre-trained timbre generation models; and inputting the timbre feature label into the first timbre generation model to generate a target vocal representation vector for the target speaker.

[0174] The timbre generation model processing module in this embodiment determines the first timbre generation model through the voice classification label, and uses the timbre feature label as input to the first timbre generation model to generate a target voice representation vector for the target speaker. That is, the timbre generation model can be used to classify and generate target voice representation vectors for different types of voices and timbres. This can make the timbre generation models of different voice classifications more accurate, thereby making the generated target voice vectors more accurate.

[0175] In an optional embodiment of the present disclosure, the above-mentioned voice classification label includes a gender classification label and an age classification label, and the timbre feature label includes a preset timbre description label.

[0176] In this embodiment, the timbre generation models of different vocal categories can be made more accurate, thereby making the generated target vocal vector more accurate.

[0177] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model is a nonlinear reversible transform flow model.

[0178] In this embodiment, the timbre generation model is a nonlinear reversible transform flow model, which can be built on the basis of normalizing flow. It can transform simple distributions (such as multidimensional prior probability distributions) into complex distributions (such as the target vocal representation vector in this embodiment), and this transformation is fully reversible. This further improves the accuracy of the timbre generation model, thereby making the generated target vocal vector more accurate.

[0179] In an optional embodiment of the present disclosure, the above-mentioned inputting the target human voice representation vector and text data into a pre-trained speech synthesis model to generate the speech data of the target speaker includes: inputting the text data into a first sub-model of the pre-trained speech synthesis model, and generating the bottleneck features of the text data through the first sub-model; inputting the target human voice representation vector and the bottleneck features into a second sub-model of the speech synthesis model, and generating the speech data of the target speaker through the second sub-model.

[0180] The speech synthesis model processing module in this embodiment can classify and generate target voice representation vectors of different types of human voices and timbres through the timbre generation model, so that the speech synthesis model can subsequently generate the speech data of the target speaker, thereby controlling the timbre characteristics of the target speaker's speech data and improving the controllability of the speaker's timbre generation.

[0181] In an optional embodiment of the present disclosure, the bottleneck features include frame-level semantic features.

[0182] In an optional embodiment of the present disclosure, the above-mentioned obtaining text data of the target speaker includes: obtaining a target text of the target speaker, and obtaining text data of the target text based on a pre-trained text processing module, wherein the text data includes phoneme data.

[0183] The label and data acquisition module in this embodiment, the text data in this embodiment may be phoneme-level data obtained through speech synthesis front-end processing, which can improve the efficiency and accuracy of acquiring the target text of the target speaker.

[0184] In an optional embodiment of the present disclosure, the process of pre-training the timbre generation model includes: obtaining a first sample vocal representation vector and a first sample annotation, and inputting the first sample vocal representation vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal representation vector; the timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys a multi-dimensional prior probability distribution; calculating a first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; taking minimizing the first loss function as the training goal, adjusting the model parameters of the timbre generation model to obtain a trained timbre generation model corresponding to the first sample annotation.

[0185] In this embodiment, when training a timbre generation model using the above module, a timbre generation model can be first identified as a target timbre generation model to be trained based on the vocal classification annotations. The timbre feature annotations are then input into the target timbre generation model to complete the training of the target timbre generation model. This makes the timbre generation models for different vocal classifications more accurate, thereby also making the generated target vocal representation vectors more accurate.

[0186] In an optional embodiment of the present disclosure, the vocal annotation of the first sample vocal representation vector includes a vocal classification annotation and a timbre feature annotation.

[0187] In an optional embodiment of the present disclosure, the above-mentioned different timbre generation models to be trained are established in advance according to the human voice classification annotations; the first sample annotation is input into the timbre generation model, including: confirming the target timbre generation model to be trained according to the human voice classification annotations; and inputting the timbre feature annotations into the target timbre generation model to be trained.

[0188] In this embodiment, a timbre generation model can be first identified based on the vocal classification annotations as the target timbre generation model to be trained, and then the timbre feature annotations can be input into the target timbre generation model to complete the training of the target timbre generation model to be trained. In this way, the timbre generation models for different vocal classifications can be made more accurate, thereby making the generated target vocal representation vector more accurate.

[0189] In an optional embodiment of the present disclosure, the multidimensional prior probability distribution is a Gaussian distribution, which has advantages such as relatively simple distribution, common use, and convenient sampling.

[0190] In an optional embodiment of the present disclosure, the above-mentioned calculation of the first loss function based on the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: calculating the relative entropy of the transformed first sample vocal representation vector and the Gaussian distribution, and using the relative entropy as the first loss function.

[0191] In an optional embodiment of the present disclosure, the above-mentioned timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; the first sample annotation is input into the timbre generation model, including: inputting the first sample annotation into the affine coupling transformation layer.

[0192] In an optional embodiment of the present disclosure, the first sample human voice representation vector and the multi-dimensional prior probability distribution have the same dimension. This embodiment can minimize the required computing resources without reducing the accuracy of the mapping.

[0193] In an optional embodiment of the present disclosure, the above-mentioned obtaining of the first sample human voice representation vector includes: obtaining a first sample human voice audio, and obtaining a first human voice representation vector of the first sample human voice audio through a trained voiceprint feature model.

[0194] In an optional embodiment of the present disclosure, the above-mentioned speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; training the second sub-model through the third sample human voice audio, so that the trained second sub-model generates the target speech according to the bottleneck feature of the human voice audio and the human voice representation vector.

[0195] The speech synthesis model in this embodiment can be divided into a first sub-model and a second sub-model. The first sub-model can be trained using a small amount of labeled data from a single speaker to predict frame-level bottleneck features from text data. The second sub-model can be trained using a large amount of unlabeled data from multiple speakers. This unlabeled data only contains audio and no corresponding text annotations. Ultimately, the first and second sub-models are cascaded to obtain the desired speech synthesis model. In this way, the second sub-model of the speech synthesis model can be trained using unlabeled multi-speaker data, thereby reducing the need for labeled training data.

[0196] In an optional embodiment of the present disclosure, the above-mentioned training of the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio includes: generating a sample text based on the second sample human voice audio; inputting the sample text into the first sub-model to obtain the first bottleneck feature; calculating the second loss function based on the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; training the first sub-model with the goal of minimizing the second loss function to obtain the trained first sub-model.

[0197] In an optional embodiment of the present disclosure, the second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; generating the sample text based on the second sample human voice audio includes: converting the second sample human voice audio into text text, and obtaining the phoneme text based on the text text.

[0198] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model through the third sample human voice audio includes: obtaining the second bottleneck feature and the second human voice representation vector of the third sample human voice audio; inputting the second bottleneck feature and the second human voice representation vector into the second sub-model, the second sub-model includes a generative adversarial network, the generative adversarial network is used to generate the target speech according to the second bottleneck feature and the second human voice representation vector, and the third sample human voice audio is used as the real sample of the generative adversarial network; with the goal of minimizing the discriminator loss function of the generative adversarial network, the second sub-model is trained to obtain the trained second sub-model.

[0199] The above method provided by the embodiment of the present disclosure can use a small amount of labeled data from a single speaker to train the first sub-model of the speech synthesis model, and use a large amount of unlabeled data from multiple speakers to train the second sub-model of the speech synthesis model. There is no need to use all labeled data to train the speech synthesis model, thereby reducing the demand for labeled training data.

[0200] In an optional embodiment of the present disclosure, the above-mentioned training of the second sub-model by the third sample human voice audio also includes: calculating the third loss function based on the generated mel spectrum of the target speech and the mel spectrum of the third sample human voice audio; training the second sub-model with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

[0201] The method provided by the embodiments of the present disclosure can use vocal labels representing the timbre characteristics of a speaker as input, thereby controlling the timbre characteristics of the speaker's speech data and improving the controllability of speaker timbre generation. Furthermore, a small amount of labeled data from a single speaker can be used to train the first sub-model of the speech synthesis model, while a large amount of unlabeled data from multiple speakers can be used to train the second sub-model of the speech synthesis model. This eliminates the need to use only labeled data to train the speech synthesis model, thus reducing the demand for labeled training data.

[0202] The voice data processing method, device, electronic device and computer program product of a readable storage medium provided in the embodiments of the present disclosure include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. For specific implementation, please refer to the method embodiment and will not be repeated here.

[0203] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0204] In addition, in the description of the embodiments of the present disclosure, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present disclosure based on the specific circumstances.

[0205] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0206] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of this disclosure and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this disclosure. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0207] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for processing speech data, the method comprising: Get the target speaker's voice label and text data; Inputting the vocal tag into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of sampling a multi-dimensional prior probability distribution and the vocal tag; The target human voice representation vector and the text data are input into a pre-trained speech synthesis model to generate speech data of the target speaker.

2. The method according to claim 1, wherein: The sampling result of sampling the multi-dimensional prior probability distribution is a multi-dimensional sampling vector.

3. The method according to claim 2, wherein: The multi-dimensional sampling vector has the same dimension as the target vocal representation vector.

4. The method according to claim 1, wherein: The vocal tag includes a vocal classification tag and a timbre feature tag; Inputting the vocal tag into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker includes: Determining a first timbre generation model corresponding to the vocal classification label according to the vocal classification label; the first timbre generation model is one of a plurality of pre-trained timbre generation models; The timbre feature label is input into the first timbre generation model to generate a target vocal representation vector of the target speaker.

5. The method according to claim 4, wherein: The vocal classification labels include gender classification labels and age classification labels, and the timbre feature labels include preset timbre description labels.

6. The method according to claim 1, wherein: The timbre generation model is a nonlinear reversible transform flow model.

7. The method according to claim 1, wherein: Inputting the target human voice representation vector and the text data into a pre-trained speech synthesis model to generate speech data of the target speaker, including: Inputting the text data into a first sub-model of a pre-trained speech synthesis model, and generating a bottleneck feature of the text data through the first sub-model; The target human voice representation vector and the bottleneck feature are input into a second sub-model of the speech synthesis model, and the speech data of the target speaker is generated through the second sub-model.

8. The method according to claim 7, wherein: The bottleneck features include frame-level semantic features.

9. The method according to claim 1, wherein: Get the target speaker's text data, including: A target text of a target speaker is obtained, and text data of the target text is obtained based on a pre-trained text processing module, wherein the text data includes phoneme data.

10. The method according to claim 1, wherein: The process of pre-training the timbre generation model includes: Obtaining a first sample vocal characterization vector and a first sample annotation, and inputting the first sample vocal characterization vector and the first sample annotation into the timbre generation model; the first sample annotation includes the vocal annotation of the first sample vocal characterization vector; The timbre generation model transforms the first sample vocal representation vector so that the transformed first sample vocal representation vector obeys the multi-dimensional prior probability distribution; Calculate a first loss function according to the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution; Taking minimization of the first loss function as a training objective, the model parameters of the timbre generation model are adjusted to obtain the timbre generation model that has been trained corresponding to the first sample annotation.

11. The method according to claim 10, wherein: The vocal annotation of the first sample vocal representation vector includes a vocal classification annotation and a timbre feature annotation.

12. The method according to claim 11, wherein: Preliminarily establishing different timbre generation models to be trained according to the vocal classification annotations; and inputting the first sample annotations into the timbre generation models, comprising: Confirming the target timbre generation model to be trained according to the vocal classification annotation; The timbre feature annotation is input into the target timbre generation model to be trained.

13. The method according to claim 10, wherein: The multidimensional prior probability distribution is a Gaussian distribution.

14. The method according to claim 13, wherein: Calculating a first loss function according to the transformed first sample vocal representation vector and the multi-dimensional prior probability distribution includes: The relative entropy between the transformed first sample vocal representation vector and the Gaussian distribution is calculated, and the relative entropy is used as the first loss function.

15. The method according to claim 10, wherein: The timbre generation model includes a nonlinear reversible transformation flow model, the flow model is a multi-layer stacked structure, and each layer of the flow model includes a normalization layer, a reversible convolution layer and an affine coupling transformation layer; Inputting the first sample annotation into the timbre generation model comprises: The first sample annotation is input into the affine coupled transformation layer.

16. The method according to claim 10, wherein: The first sample vocal characterization vector has the same dimension as the multi-dimensional prior probability distribution.

17. The method according to claim 10, wherein: Obtaining the first sample vocal representation vector, including: A first sample human voice audio is obtained, and a first human voice representation vector of the first sample human voice audio is obtained through a trained voiceprint feature model.

18. The method according to claim 1, wherein: The speech synthesis model includes a first sub-model and a second sub-model, and the process of pre-training the speech synthesis model includes: Obtaining a second sample human voice audio, a bottleneck feature annotation of the second sample human voice audio, and a third sample human voice audio; Training the first sub-model through the second sample human voice audio and the bottleneck feature annotation of the second sample human voice audio, so that the trained first sub-model obtains the bottleneck feature according to the human voice audio; The second sub-model is trained by using the third sample vocal audio so that the trained second sub-model generates the target speech according to the bottleneck features of the vocal audio and the vocal representation vector.

19. The method according to claim 18, wherein: Training the first sub-model by annotating the bottleneck features of the second sample human voice audio and the second sample human voice audio includes: Generate a sample text according to the second sample human voice audio; Inputting the sample text into the first sub-model to obtain a first bottleneck feature; Calculate a second loss function according to the first bottleneck feature and the bottleneck feature annotation of the second sample human voice audio; The first sub-model is trained with the goal of minimizing the second loss function to obtain a trained first sub-model.

20. The method according to claim 19, wherein: The second sample human voice audio is a phoneme annotation, and the sample text is a phoneme text; and generating the sample text according to the second sample human voice audio includes: The second sample human voice audio is converted into text, and the phoneme text is obtained according to the text.

21. The method according to claim 18, wherein: Training the second sub-model by using the third sample human voice audio includes: Obtaining a second bottleneck feature and a second voice representation vector of the third sample voice audio; Inputting the second bottleneck feature and the second vocal representation vector into the second sub-model, wherein the second sub-model includes a generative adversarial network, wherein the generative adversarial network is used to generate a target speech according to the second bottleneck feature and the second vocal representation vector, and the third sample vocal audio is used as a real sample of the generative adversarial network; The second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network to obtain a trained second sub-model.

22. The method according to claim 21, wherein: The second sub-model is trained by the third sample human voice audio, further comprising: Calculate a third loss function based on the generated Mel spectrum of the target speech and the Mel spectrum of the third sample human voice audio; The second sub-model is trained with the goal of minimizing the discriminator loss function of the generative adversarial network and the third loss function.

23. A device for processing speech data, the device comprising: A label and data acquisition module, configured to execute acquisition of a target speaker's voice label and text data; A timbre generation model processing module is configured to input the vocal tag into a pre-trained timbre generation model to generate a target vocal representation vector of the target speaker, wherein the timbre generation model generates the target vocal representation vector based on a sampling result of sampling a multi-dimensional prior probability distribution and the vocal tag; The speech synthesis model processing module is configured to input the target human voice representation vector and the text data into a pre-trained speech synthesis model to generate speech data of the target speaker.

24. An electronic device comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the voice data processing method described in any one of claims 1-22.

25. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method for processing speech data according to any one of claims 1 to 22.

Citation Information

Patent Citations

  • Speech synthesis method and device

    CN112002302A

  • Multi-language speech synthesis model training method and device

    CN113707125A

  • Cross-speaker voice style modeling method and computer readable storage medium

    CN114242031A

  • Speech synthesis method and device, medium, computer equipment and program product

    CN114360493A

  • Voice data processing method and device, electronic equipment and readable storage medium

    CN116564269A

Cited By

  • Speech synthesis method and device, equipment and storage medium

    CN119811363A