A method and system for converting speech of an unknown speaker
By constructing a customized speech synthesis model and utilizing adaptive encoders and decoders to control text features, the problem of requiring a large amount of data for generating speech from unknown speakers in existing technologies has been solved, achieving efficient and natural speech conversion results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN OCTOPUS TECH
- Filing Date
- 2023-02-02
- Publication Date
- 2026-05-26
AI Technical Summary
Existing speech conversion methods require a large amount of data for training when generating speech from unknown speakers, and the naturalness and similarity of the synthesized speech are poor.
By using a small amount of unknown speaker speech data and making minor changes to the speech synthesis model parameters, and by utilizing the text feature modulation within the adaptive posterior encoder, adaptive encoder, and adaptive stream decoder, a customized speech synthesis model is constructed to quickly adapt to the speaker's timbre.
It enables the generation of high-quality, highly similar speech from unknown speakers under limited data conditions, improving the efficiency and naturalness of speech synthesis.
Smart Images
Figure CN116504219B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech synthesis technology, and more specifically to a method and system for converting the speech of an unknown speaker. Background Technology
[0002] In existing speech conversion methods, many methods use decoupling to extract speaker features and text features, but the synthesized speech does not have good naturalness and similarity. Therefore, a speech conversion method based on speech feature extraction model and speech synthesis model can generate converted speech more effectively. The speech feature extraction model extracts text features from the input speech, links them to the target speaker's voice trained by the speech synthesis model, reconstructs the target Mel spectrum (acoustic features), and feeds the target Mel spectrum into the vocoder model of speech synthesis to obtain the target speech.
[0003] However, the aforementioned speech conversion methods based on speech feature extraction models and speech synthesis models have the drawback of requiring a large amount of data for training when trying to acquire new speaker voices. Summary of the Invention
[0004] To address the aforementioned problems, one objective of this invention is to provide a method for converting the speech of an unknown speaker. This method uses a small amount of unknown speaker speech data and converts the unknown speaker's voice by making minor changes to the parameters of the speech synthesis model. This method uses the speaker's high-dimensional features as conditions to regulate the text features within the adaptive posterior encoder, adaptive encoder, and adaptive stream decoder, thereby achieving a faster and better technical effect of adapting to the speaker's timbre.
[0005] A second objective of this invention is to provide a system for converting the speech of an unknown speaker.
[0006] The first technical solution adopted in this invention is: a method for converting the speech of an unknown speaker, comprising the following steps:
[0007] S100: Acquire voice data, wherein the voice data is labeled with speaker tags;
[0008] S200: Preprocess the speech data to obtain text features, linear spectrum, and speech waveform;
[0009] S300: Construct a speech synthesis model, and perform adversarial pre-training on the speech synthesis model based on the speaker labels, text features, linear spectrum and speech waveform to form a pre-trained speech synthesis model;
[0010] S400: Fine-tune the pre-trained speech synthesis model to generate a customized speech synthesis model;
[0011] S500: Obtain the speech of an unknown speaker based on the customized speech synthesis model.
[0012] Preferably, step S200 includes the following sub-steps:
[0013] S210: Use a text feature extractor to extract text features for each piece of speech data;
[0014] S220: Perform pre-emphasis, framing, windowing, and short-time Fourier transform processing on each piece of speech data to obtain a linear spectrum;
[0015] S230: Each piece of voice data is stored in the form of discrete points, which forms a voice waveform.
[0016] Preferably, the speech synthesis model in step S300 includes an acoustic model, a vocoder, and a speaker feature table. The acoustic model includes an adaptive encoder, an adaptive stream decoder, an adaptive posterior encoder, a waveform generator, and a discriminator.
[0017] Preferably, the adaptive encoder is composed of four stacked AFT modules, each AFT module comprising an adaptive attention module, a convolutional network, a residual connection, and a CLN layer.
[0018] Preferably, both the adaptive stream decoder and the adaptive posterior encoder are composed of AFlow modules, and the AFlow module includes an adaptive affine coupling layer; the adaptive affine coupling layer is obtained by adding a CLN layer to the network that predicts the parameters of the affine coupling layer.
[0019] Preferably, the waveform generator and the discriminator adopt the architecture of a HiFi-GAN vocoder; the discriminator includes a multi-period discriminator and a multi-scale discriminator.
[0020] Preferably, the adversarial pre-training in step S300 includes the following sub-steps:
[0021] S310: Query the speaker feature table based on the speaker label to generate speaker features;
[0022] S320: Input the text features and speaker features into the adaptive encoder to generate prior mean and variance;
[0023] S330: Input the linear spectrum and speaker features into the adaptive posterior encoder to generate the posterior mean and variance; generate Gaussian noise, and obtain posterior features based on the posterior mean, variance and Gaussian noise;
[0024] S340: Input the posterior features into the adaptive stream decoder and obtain the posterior stream features through encoding;
[0025] S350: Input the posterior features into the waveform generator to generate a predicted speech waveform; input the predicted speech waveform and the speech waveform into the discriminator to obtain a discrimination result;
[0026] S360: The loss is calculated using the identification results, posterior flow features, prior mean and variance, and the speech synthesis model parameters are updated to achieve resistive pre-training of the speech synthesis model.
[0027] Preferably, step S400 includes:
[0028] Obtain the speech of an unknown speaker, freeze all weights in the pre-trained speech synthesis model except for the CLN layer weights and the speaker feature table, randomly initialize the speaker features of the unknown speaker, repeat the above pre-training steps, and fine-tune the pre-trained speech synthesis model to form a customized speech synthesis model.
[0029] Preferably, step S500 includes:
[0030] Input the speech to be converted and the corresponding speaker labels. The speech to be converted is used to extract text features through a text feature extractor. The corresponding speaker labels are mapped to speaker features through a speaker feature table.
[0031] The text features and speaker features are input into the adaptive encoder in the customized speech synthesis model to generate prior mean and variance; Gaussian noise is generated, and the prior mean, variance, and Gaussian noise are input into the adaptive stream decoder in the customized speech synthesis model. After encoding, the noise is input into the waveform generator in the customized speech synthesis model to generate a speech waveform, which is converted into the speech of an unknown speaker.
[0032] The second technical solution adopted in this invention is: a system for converting the speech of an unknown speaker, comprising a data acquisition module, a preprocessing module, a pre-training module, a fine-tuning training module, and a speech conversion module.
[0033] The data acquisition module is used to acquire voice data, which is labeled with speaker tags.
[0034] The preprocessing module is used to preprocess the speech data to obtain text features, linear spectrum, and speech waveform;
[0035] The pre-training module is used to construct a speech synthesis model. Based on the text features, linear spectrum and speech waveform, the speech synthesis model is pre-trained adversarially to form a pre-trained speech synthesis model.
[0036] The fine-tuning training module is used to fine-tune the pre-trained speech synthesis model to generate a customized speech synthesis model.
[0037] The speech conversion module is used to obtain the speech of an unknown speaker based on the customized speech synthesis model.
[0038] The beneficial effects of the above technical solution are as follows:
[0039] (1) The present invention discloses a method for converting the speech of an unknown speaker. It uses a small amount of speech data of an unknown speaker and converts the speech of an unknown speaker by making small changes to the parameters of the speech synthesis model. The method uses the high-dimensional features of the speaker as a condition to regulate the text features inside the adaptive posterior encoder, adaptive encoder and adaptive stream decoder, so as to achieve a faster and better technical effect of adapting to the speaker's timbre. Attached Figure Description
[0040] Figure 1 A flowchart illustrating a method for converting the speech of an unknown speaker, provided as an embodiment of the present invention;
[0041] Figure 2 A flowchart illustrating a method for converting the speech of an unknown speaker, as provided in one embodiment of the present invention;
[0042] Figure 3 This is an architecture diagram of a speech synthesis model provided in one embodiment of the present invention;
[0043] Figure 4 This is a schematic diagram of the structure of an AFT provided in one embodiment of the present invention;
[0044] Figure 5 This is a schematic diagram of the structure of the AFlow module provided in one embodiment of the present invention;
[0045] Figure 6 This is a schematic diagram of the structure of an adaptive affine coupling layer provided in one embodiment of the present invention;
[0046] Figure 7 This is a schematic diagram illustrating the process of obtaining speech from an unknown speaker based on a customized speech synthesis model, as provided in one embodiment of the present invention.
[0047] Figure 8 This is a schematic diagram of the structure of a system for converting the speech of an unknown speaker, provided as an embodiment of the present invention. Detailed Implementation
[0048] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.
[0049] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.
[0050] Example 1
[0051] like Figure 1 and Figure 2 As shown, one embodiment of the present invention provides a method for converting the speech of an unknown speaker, comprising the following steps:
[0052] S100: Acquire voice data, wherein the voice data is labeled with speaker tags;
[0053] Record speech data from several speakers, with each speaker recording a sufficient amount of speech data (e.g., 300-400 sentences, each sentence lasting approximately 5-6 seconds) to serve as the training dataset; each sentence is treated as a speech data point and is labeled with a speaker tag, such as a speaker number or index; the number of speakers can be determined as needed, for example, set to 22 speakers.
[0054] S200: Preprocess the speech data to obtain text features, linear spectrum, and speech waveform;
[0055] S210: Use a text feature extractor to extract text features for each piece of speech data;
[0056] For each speech data in the training dataset, the text features of each speech data are extracted using a text feature extractor. This text feature extractor can be any suitable unsupervised or supervised speech representation learning model, such as HuBERT, Wav2vec, etc.
[0057] S220: Perform pre-emphasis, framing, windowing, and short-time Fourier transform processing on each piece of speech data to obtain a linear spectrum;
[0058] Each speech data point is processed through pre-emphasis, framing, windowing, and short-time Fourier transform to obtain a linear spectrum. The relevant parameters for speech data processing are shown in Table 1, but are not limited to these and can be adjusted appropriately according to the actual situation.
[0059] Table 1 Audio Parameters
[0060] Parameter name value Sampling rate 16000HZ Frame length 1280 points Window length 1280 points Frame shift 320 points Pre-weighting coefficient 0.97
[0061] S230: Generate text feature-linear spectrum-speech waveform data pairs based on each speech data and the obtained text features and linear spectrum.
[0062] Each piece of voice data is stored in the computer as discrete points; that is, the voice waveform is the voice data.
[0063] S300: Construct a speech synthesis model (i.e., a speech synthesis module), and perform adversarial pre-training on the speech synthesis model based on speaker labels, text features, linear spectrum and speech waveform to form a pre-trained speech synthesis model;
[0064] (1) Construct a speech synthesis model;
[0065] like Figure 3 As shown, the speech synthesis model includes an acoustic model and a vocoder. The acoustic model includes an adaptive encoder (AFT*4), an adaptive stream decoder (AFlow*4), an adaptive posterior encoder (AFlow*1), a waveform generator (HiFi-GANgenerator), and a discriminator (HiFi-GAN discriminator).
[0066] The speech synthesis model also includes a speaker feature table, which contains 256 hidden units.
[0067] like Figure 4 As shown, the adaptive encoder consists of four stacked adaptive forward transform (AFT) modules. The AFT structure includes, in sequence, an adaptive-self-attention (ASA) module with 256 hidden units each, a two-layer 1D convolutional network, a residual connection, and a CLN layer with 256 hidden units. The hidden units are all 1024, the kernel sizes are 3 and 9 respectively, and the stride is 1. The ReLU activation function is used inside the ASA module and inside the two-layer convolutional network.
[0068] Both the adaptive stream decoder and the adaptive posterior encoder are constructed using Adaptive Flow (AFlow) modules. The adaptive stream decoder uses four stacked AFlow modules, while the adaptive posterior encoder uses one AFlow module. Figure 5As shown, the AFlow module includes an adaptive affine coupling layer; the affine coupling layer is a traditional flow module, while the adaptive affine coupling layer adds a CLN layer to the network of its prediction parameters. Its specific structure is as follows. Figure 6 As shown, the input is a matrix with hidden_num rows and length columns. Input_a and input_b divide this matrix into two parts in terms of dimensions; Input_a uses the upper half of the matrix (i.e., hidden_num / 2 channels), and input_b uses the other half; concat is the concatenation function, which concatenates the input into a shape [hidden_num, length]; for example, hidden_num represents the number of features, such as =128, and length represents the length of the input sequence, such as =27, that is, the input is a 128*27 matrix.
[0069] The waveform generator and discriminator adopt the architecture of HiFi-GAN vocoder. Compared with HiFi-GAN, which uses the mel spectrum as input, this invention uses the latent variable Z as input feature. The latent variable Z is generated by the adaptive posterior encoder during the training phase, and is obtained by the adaptive stream decoder during the testing and actual operation phases.
[0070] The discriminator (HiFi-GAN discriminator) includes a multi-period discriminator (MPD) and a multi-scale discriminator (MSD). The multi-period discriminator can have 5 different discriminators based on the period [2, 3, 5, 7, 11], so there are a total of 6 discriminators. These include 5 discriminators with different periods and 1 multi-scale discriminator. The discriminator feeds the predicted speech waveform and the speech waveform into these 6 discriminators in parallel and obtains the corresponding results, and then performs loss calculation. This invention uses the input predicted speech waveform and speech waveform, and the discriminator to determine whether the object is real or fake.
[0071] The MSD multi-scale discriminator has 6 layers of 1D convolutions and a leakyRelu() activation function, with the following parameters:
[0072] Conv1d(channels=16,kernal_size=15,stride=1)
[0073] leakyRelu()
[0074] Conv1d(channels=64,kernal_size=41,stride=4)
[0075] leakyRelu()
[0076] Conv1d(channels=256,kernal_size=41,stride=4)
[0077] leakyRelu()
[0078] Conv1d(channels=1024,kernal_size=41,stride=4)
[0079] leakyRelu()
[0080] Conv1d(channels=1024,kernal_size=41,stride=4)
[0081] leakyRelu()
[0082] Conv1d(channels=1024,kernal_size=5,stride=1)
[0083] leakyRelu()
[0084] Each time a Conv1d layer is passed, it goes through the leakyRelu() activation function once.
[0085] The model structure of the MPD multi-cycle discriminator is completely the same, but its input is transformed from a 1-dimensional waveform sequence to a 2-dimensional matrix. The transformation method can be to transform the 1-dimensional waveform into a matrix with [2,3,5,7,11] points as a group.
[0086] The structure of MPD is no longer 1D convolution, but 2D convolution, with a total of 5 layers of 2D convolution and the corresponding leaky_Relu() activation function, the parameters of which are as follows:
[0087] Conv2d(channel=32,kernal_size=(5,1),stride=(3,1))
[0088] leakyRelu()
[0089] Conv2d(channel=128,kernal_size=(5,1),stride=(3,1))
[0090] leakyRelu()
[0091] Conv2d(channel=512,kernal_size=(5,1),stride=(3,1))
[0092] leakyRelu()
[0093] Conv2d(channel=1024,kernal_size=(5,1),stride=(3,1))
[0094] leakyRelu()
[0095] Conv2d(channel=1024,kernal_size=(5,1),stride=(3,1))
[0096] leakyRelu().
[0097] (2) Adversarial pre-training of the speech synthesis model based on speaker labels, text features, linear spectrum, and speech waveform, including:
[0098] S310: Query the speaker feature table based on the speaker label to generate speaker feature S;
[0099] S320: Input text features and speaker features into the adaptive encoder to generate prior mean and variance;
[0100] S330: Input the linear spectrum and speaker features into the adaptive posterior encoder to generate the posterior mean and variance; generate Gaussian noise, and obtain posterior features based on the posterior mean, variance, and Gaussian noise; the Gaussian noise is generated by a correlation function, such as the torch.randn function with 0 mean and 1 variance.
[0101] S340: Input the posterior features into the adaptive stream decoder and obtain the posterior stream features through encoding;
[0102] S350: Input the posterior features into the waveform generator to generate the predicted speech waveform; input the predicted speech waveform and the speech waveform into the discriminator to obtain the discrimination result;
[0103] S360: It calculates the loss based on the identification results, posterior flow features, prior mean and variance, and updates the speech synthesis model parameters based on the loss, thereby achieving anti-pre-training of the speech synthesis model.
[0104] In the pre-training phase, this invention uses a batch size of 32 and the AdamW optimizer with parameters betas: [0.8, 0.99] and eps = 1e-9 (10 to the power of -9). For the waveform generator G and the discriminator D, this invention uses two optimizers and performs adversarial (GAN) training steps. Part of the data used in step 6) above is used for the loss function Loss(D) of the discriminator D, and part is used for the loss function Loss(G) of the waveform generator G. That is, the discriminator is first optimized once using the loss function Loss(D) of the discriminator D in the speech synthesis module, and then the generator G is optimized once using the loss function Loss(G) of the waveform generator G. Each step is composed of these two small... Step completed; initial learning rate is lr = 2e-4, iterative training for 300 epochs; due to the gradient backpropagation characteristics during model training, although Loss(G) is used for the waveform generator, it updates the parameters of all modules below it (i.e., modules in the speech synthesis model except for all discriminators) when updating parameters, while Loss(D) only updates the discriminator itself through technical means.
[0105] S400: Fine-tunes the pre-trained speech synthesis model to generate a customized speech synthesis model;
[0106] Obtain 10-20 unknown speaker voices, freeze all weights in the pre-trained speech synthesis model except for the CLN layer weights and the speaker feature table, randomly initialize the speaker features of the unknown speakers, repeat the above pre-training steps, and fine-tune the pre-trained speech synthesis model (keeping the soft-HuBERT unchanged); after 2000 steps of fine-tuning, a customized speech synthesis model is formed.
[0107] Furthermore, in one embodiment, a customized speech synthesis model is tested;
[0108] Select 10-20 local speakers for testing; subjective evaluation will be conducted based on both speech quality (MOS) and speaker similarity (SMOS); a standard international 5-point scoring system will be used, ranging from 0 to 5.
[0109] Very poor: The emotions expressed are completely unrelated to the target emotions, and the emotional expression is extremely poor;
[0110] Very poor: The emotions are basically consistent with the target emotions, but the emotional expression is extremely poor;
[0111] Medium: The emotions are relatively close to the target emotions, and the emotional expression is good;
[0112] Good: The emotions are relatively close to the target emotions, and the emotional expression is sufficient;
[0113] Excellent: The emotions are closely aligned with the target emotions, and the emotional expression is outstanding; each interval is 0.5 points.
[0114] The test results are shown in Table 2.
[0115] Table 2 Test data for the customized speech synthesis model
[0116] MOS SMOS GT (Real Voice) 3.96 3.87 The customized speech synthesis model of the present invention 3.78 3.64
[0117] S500: Obtains the speech of an unknown speaker based on a customized speech synthesis model.
[0118] like Figure 7 As shown, the input is a speech to be converted and the corresponding speaker index (i.e., the target speaker label). The speech to be converted is processed by a text feature extractor to extract text features; the corresponding target speaker index is mapped to speaker features through a speaker feature table.
[0119] Text features and speaker features are input into the adaptive encoder in the customized speech synthesis model to generate prior mean and variance; Gaussian noise is generated, and the prior mean, variance, and Gaussian noise are input into the adaptive stream decoder. After encoding, the noise is input into the waveform generator to generate a speech waveform, which is the speech of the unknown speaker.
[0120] Example 2
[0121] like Figure 8 As shown, one embodiment of the present invention provides a system for converting the speech of an unknown speaker, including a data acquisition module, a preprocessing module, a pre-training module, a fine-tuning training module, and a speech conversion module:
[0122] The data acquisition module is used to acquire voice data, which is labeled with speaker tags.
[0123] The preprocessing module is used to preprocess the speech data to obtain text features, linear spectrum, and speech waveform;
[0124] The pre-training module is used to construct a speech synthesis model. Based on the text features, linear spectrum and speech waveform, the speech synthesis model is pre-trained adversarially to form a pre-trained speech synthesis model.
[0125] The fine-tuning training module is used to fine-tune the pre-trained speech synthesis model to generate a customized speech synthesis model.
[0126] The speech conversion module is used to obtain the speech of an unknown speaker based on the customized speech synthesis model.
[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0129] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0130] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0131] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method of converting speech of an unknown speaker, characterized by, Includes the following steps: S100: Acquire voice data, wherein the voice data is labeled with speaker tags; S200: Preprocess the speech data to obtain text features, linear spectrum, and speech waveform; S300: Construct a speech synthesis model, which includes an acoustic model and a speaker feature table. The acoustic model includes an adaptive encoder, an adaptive stream decoder, an adaptive posterior encoder, a waveform generator, and a discriminator. Based on the speaker labels, text features, linear spectrum, and speech waveform, perform adversarial pre-training on the speech synthesis model to form a pre-trained speech synthesis model. S400: Fine-tune the pre-trained speech synthesis model to generate a customized speech synthesis model; S5 00: Obtain the speech of an unknown speaker based on the customized speech synthesis model; The adversarial pre-training in step S300 includes the following sub-steps: S310: Based on the speaker label, query the speaker feature table to generate speaker features; S320: Input the text features and speaker features into the adaptive encoder to generate prior mean and variance; S330: Input the linear spectrum and speaker features into the adaptive posterior encoder to generate the posterior mean and variance; generate Gaussian noise, and obtain posterior features based on the posterior mean, variance and Gaussian noise; S340: Input the posterior features into the adaptive stream decoder and obtain the posterior stream features through encoding; S350: Input the posterior features into the waveform generator to generate a predicted speech waveform; input the predicted speech waveform and the speech waveform into the discriminator to obtain a discrimination result; S360: The loss is calculated using the identification results, posterior flow features, prior mean and variance, and the speech synthesis model parameters are updated to achieve resistive pre-training of the speech synthesis model.
2. The method of converting speech of an unknown speaker according to claim 1, wherein, Step S200 includes the following sub-steps: S210: Use a text feature extractor to extract text features for each piece of speech data; S220: Perform pre-emphasis, framing, windowing, and short-time Fourier transform processing on each piece of speech data to obtain a linear spectrum; S230: Each piece of voice data is stored in the form of discrete points, which forms a voice waveform.
3. The method of converting speech of an unknown speaker according to claim 1, wherein, The adaptive encoder is composed of four stacked AFT modules, each of which includes an adaptive attention module, a convolutional network, a residual connection, and a CLN layer.
4. The method for converting speech of an unknown speaker according to claim 1, characterized in that, Both the adaptive stream decoder and the adaptive posterior encoder are composed of AFlow modules, which include an adaptive affine coupling layer. The adaptive affine coupling layer is obtained by adding a CLN layer to the network that predicts the parameters of the affine coupling layer.
5. The method for converting speech of an unknown speaker according to claim 1, characterized in that, The waveform generator and the discriminator adopt the architecture of HiFi-GAN vocoder; the discriminator includes a multi-period discriminator and a multi-scale discriminator.
6. The method for converting speech of an unknown speaker according to claim 1, characterized in that, Step S500 includes: Input the speech to be converted and the corresponding speaker labels. The speech to be converted is used to extract text features through a text feature extractor. The corresponding speaker labels are mapped to speaker features through a speaker feature table. The text features and speaker features are input into the adaptive encoder in the customized speech synthesis model to generate prior mean and variance; Gaussian noise is generated, and the prior mean, variance, and Gaussian noise are input into the adaptive stream decoder in the customized speech synthesis model. After encoding, the noise is input into the waveform generator in the customized speech synthesis model to generate a speech waveform, which is converted into the speech of an unknown speaker.
7. A system for converting the speech of an unknown speaker, characterized in that, It includes a data acquisition module, a preprocessing module, a pre-training module, a fine-tuning training module, and a speech conversion module. The data acquisition module is used to acquire voice data, which is labeled with speaker tags. The preprocessing module is used to preprocess the speech data to obtain text features, linear spectrum, and speech waveform; The pre-training module is used to construct a speech synthesis model. Based on the text features, linear spectrum, and speech waveform, the speech synthesis model is pre-trained adversarially to form a pre-trained speech synthesis model. The speech synthesis model includes an acoustic model and a speaker feature table. The acoustic model includes an adaptive encoder, an adaptive stream decoder, an adaptive posterior encoder, a waveform generator, and a discriminator. The pre-training module is used to perform the following operations: S310: Based on the speaker label, query the speaker feature table to generate speaker features; S320: Input the text features and speaker features into the adaptive encoder to generate prior mean and variance; S330: Input the linear spectrum and speaker features into the adaptive posterior encoder to generate the posterior mean and variance; generate Gaussian noise, and obtain posterior features based on the posterior mean, variance and Gaussian noise; S340: Input the posterior features into the adaptive stream decoder and obtain the posterior stream features through encoding; S350: Input the posterior features into the waveform generator to generate a predicted speech waveform; input the predicted speech waveform and the speech waveform into the discriminator to obtain a discrimination result; S360: The loss is calculated using the identification results, posterior flow features, prior mean and variance, and the speech synthesis model parameters are updated to achieve resistive pre-training of the speech synthesis model. The fine-tuning training module is used to fine-tune the pre-trained speech synthesis model to generate a customized speech synthesis model. The speech conversion module is used to obtain the speech of an unknown speaker based on the customized speech synthesis model.