Feature decoupling model training, speech decoupling, speech synthesis method and device

Abstract

The present application relates to artificial intelligence technology, and proposes a method and device for training a feature decoupling model, speech decoupling, and speech synthesis, the method comprising: inputting a first spectral feature of a sample audio into a rhythm encoder of a pre-trained speech feature decoupling model to encode the rhythm code; using the model's content encoder to encode the first spectral feature to obtain the content code; using the model's pitch encoder to encode the pitch contour feature to obtain the pitch code; using the model's decoder to decode the rhythm code, content code, pitch code, and user identity feature input into the decoder to obtain a second spectral feature; obtaining an overall loss function based on the calculated reconstruction loss function and orthogonal loss function; calculating the gradient based on the overall loss function to iteratively update the model parameters until the convergence condition is reached, and obtaining a trained speech feature decoupling model. The present application can optimize speech feature decoupling to achieve the purpose of full decoupling.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for training a feature decoupling model, speech decoupling, and speech synthesis. Background Art

[0002] Voice conversion technology is one of the speech fields that has attracted the attention of researchers around the world. Voice conversion is widely used in many fields, such as non-parallel many-to-many VC, such as deep style transfer algorithm, generative adversarial network (GAN) and conditional variational autoencoder (CVAE).

[0003] The basis of speech conversion is to decouple and reorganize speech component features. Speech decoupling is to split the original speaker's speech into component features or audio components of multiple dimensions, such as content vector, timbre vector, pitch vector and rhythm vector. Reorganization means, for example, specifying content vector, pitch vector and rhythm vector, and given any timbre vector, it can be converted into the speech of any speaker.

[0004] Existing technologies can only decouple content and timbre, but pitch, rhythm and content are still mixed, and it is impossible to completely decouple the various component features of the speaker's voice. In addition, in order to further unravel the remaining voice component features, relevant models need to be effectively trained. However, training samples do not have clear labels, and these labels are expensive to obtain. Summary of the invention

[0005] In order to solve the technical problem that the various component features in speech cannot be completely decoupled in the prior art, the present application provides a method and device for training a feature decoupling model, speech decoupling, and speech synthesis, the main purpose of which is to achieve complete decoupling of various components of speech.

[0006] To achieve the above objectives, the present application provides a method for training a speech feature decoupling model, the method comprising:

[0007] Obtaining a first spectrum feature of the sample audio and a pitch contour feature obtained after normalization;

[0008] Inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code;

[0009] Performing a second encoding on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model to obtain a content code;

[0010] Performing a third encoding on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch encoding;

[0011] Decoding the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in the pre-trained speech feature decoupling model to obtain a second spectrum feature;

[0012] Calculating a reconstruction loss function according to the first spectrum feature and the second spectrum feature;

[0013] Calculating an orthogonal loss function based on correlations between pairs of audio components, wherein each pair of audio components includes two different audio components, and the audio components include rhythm coding, pitch coding, and content coding, or the audio components include rhythm coding, pitch coding, timbre coding, and content coding;

[0014] The overall loss function of the pre-trained speech feature decoupling model is obtained according to the reconstruction loss function and the orthogonal loss function;

[0015] The gradient is calculated according to the overall loss function, and the model parameters of the pre-trained speech feature decoupling model are iteratively updated according to the gradient until the convergence condition is reached, thereby obtaining the trained speech feature decoupling model.

[0016] In addition, to achieve the above purpose, the present application also provides a method for decoupling speech features, the method comprising:

[0017] Obtaining target spectral features of the speech to be decoupled and target pitch contour features obtained after normalization;

[0018] The target spectral features are input into the rhythm encoder and content encoder of the trained speech feature decoupling model obtained according to any of the previous items, and the target pitch contour features are input into the pitch encoder of the trained speech feature decoupling model for encoding and decoupling respectively to obtain the target rhythm code, target content code and target pitch code.

[0019] In addition, to achieve the above purpose, the present application also provides a speech synthesis method, which includes:

[0020] Acquire a third spectrum feature of the first speech and a second pitch contour feature obtained after normalization;

[0021] Acquire a fourth frequency spectrum feature of the second speech and a third pitch contour feature obtained after normalization;

[0022] Inputting one of the third spectral feature and the fourth spectral feature into a rhythm encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target rhythm encoding;

[0023] Inputting one of the third spectral feature and the fourth spectral feature into a content encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target content encoding;

[0024] Inputting one of the second pitch contour feature and the third pitch contour feature into a pitch encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target pitch encoding;

[0025] Inputting a target user identity feature selected from a first user identity feature corresponding to a first voice and a second user identity feature corresponding to a second voice, as well as a target rhythm code, a target content code, and a target pitch code obtained, into a decoder of a trained voice feature decoupling model obtained according to any of the preceding items for decoding to obtain a synthesized target spectral feature, wherein the inputs of a rhythm encoder, a content encoder, and a pitch encoder in the trained voice feature decoupling model and the target user identity feature input into the decoder of the trained voice feature decoupling model are not features corresponding to the same voice;

[0026] The target spectral features are converted into target synthesized speech using a vocoder.

[0027] In addition, to achieve the above purpose, the present application also provides a training device for a speech feature decoupling model, the device comprising:

[0028] A data acquisition module, used to acquire a first frequency spectrum feature of the sample audio and a pitch contour feature obtained after standardization;

[0029] A rhythm encoding module, used for inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code;

[0030] A content encoding module, used to perform a second encoding on the first spectral feature using a content encoder included in a pre-trained speech feature decoupling model to obtain a content code;

[0031] A pitch encoding module, used for performing a third encoding on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch code;

[0032] A decoding module, used for decoding the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in the pre-trained speech feature decoupling model to obtain a second spectrum feature;

[0033] A first calculation module, used for calculating a reconstruction loss function according to the first spectrum feature and the second spectrum feature;

[0034] A second calculation module is used to calculate an orthogonal loss function according to the correlation between every two audio components, wherein the audio components include rhythm coding, pitch coding and content coding, or the audio components include rhythm coding, pitch coding, timbre coding and content coding;

[0035] A third calculation module is used to obtain an overall loss function of the pre-trained speech feature decoupling model according to the reconstruction loss function and the orthogonal loss function;

[0036] The parameter updating module is used to calculate the gradient according to the overall loss function, and iteratively update the model parameters of the pre-trained speech feature decoupling model according to the gradient until the convergence condition is reached, thereby obtaining the trained speech feature decoupling model.

[0037] To achieve the above-mentioned objectives, the present application also provides a computer device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the steps of the training method of the speech feature decoupling model as described in any of the foregoing items are executed, or, when the processor executes the computer-readable instructions, the steps of the speech feature decoupling method as described in any of the foregoing items are executed, or, when the processor executes the computer-readable instructions, the steps of the speech synthesis method as described in any of the foregoing items are executed.

[0038] To achieve the above-mentioned objectives, the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor executes the steps of the training method of the speech feature decoupling model as described in any of the foregoing items, or the processor executes the steps of the speech feature decoupling method as described in any of the foregoing items, or the processor executes the steps of the speech synthesis method as described in any of the foregoing items.

[0039] The method and device for training the feature decoupling model, speech decoupling, and speech synthesis proposed in the present application, by constructing a speech feature decoupling model, and training the rhythm encoder, pitch encoder, content encoder, and decoder contained in the speech feature decoupling model, calculating the overall loss function by reconstructing the loss function and the orthogonal loss function, iteratively optimizing the model parameters according to the overall loss function for the purpose of minimizing the overall loss function as much as possible, so as to further optimize the decoupling ability of the speech feature decoupling model, so that the speech feature decoupling model obtained after training can fully decouple the speech, and each component feature is independently decomposed. In addition, the training samples of this embodiment do not require complex sample labels, and the sample labels are easy to obtain. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a method for training a speech feature decoupling model in an embodiment of the present application;

[0041] Figure 2 This is a structural block diagram of a speech feature decoupling model in an embodiment of the present application;

[0042] Figure 3 This is a structural block diagram of a training device for a speech feature decoupling model in one embodiment of the present application;

[0043] Figure 4 This is a block diagram of the internal structure of a computer device in one embodiment of the present application.

[0044] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of this application. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Figure 1 FIG. 1 is a flow chart of a method for training a speech feature decoupling model in an embodiment of the present application; Figure 1 The training method of the speech feature decoupling model includes the following steps S100-S900.

[0047] S100: Acquire a first frequency spectrum feature of a sample audio and a pitch contour feature obtained after standardization.

[0048] Specifically, the first spectrum feature is a Mel-spectrogram (Mel spectrum or Mel spectrogram) obtained by transforming the sample audio. The first spectrum feature includes the rhythm feature (Rhythm), pitch feature (Pitch), content feature (Content) and timbre feature (Timbre) contained in the sample audio. The pitch contour feature is a pitch contour that standardizes the pitch to a uniform mean and a uniform variance. The pitch contour feature only includes partial rhythm features (incomplete rhythm features) and pitch features of the audio.

[0049] The structural block diagram of the speech feature decoupling model in the training method of the speech feature decoupling model provided in this application is as follows: Figure 2 Reference Figure 2 ,The speech feature decoupling model is a generative model based on an autoencoder, including a rhythm encoder, a content encoder, a pitch encoder and a decoder.

[0050] S200: Inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code.

[0051] S300: Perform a second encoding on the first spectral feature using a content encoder included in a pre-trained speech feature decoupling model to obtain a content code.

[0052] S400: Perform a third encoding on the pitch contour feature using the pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch code.

[0053] Specifically, in a specific embodiment, a corresponding information bottleneck is added to the rhythm encoder, the content encoder, and the pitch encoder. The information bottleneck forces the encoder to only pass features or information that other encoders cannot provide and discard other features. The size of the information bottleneck added by each encoder may be different and will be adjusted according to the corresponding rule settings.

[0054] The rhythm encoder is used to extract the rhythm code from the first spectrum feature. The content encoder is used to extract the content code from the first spectrum feature. The pitch encoder is used to extract the pitch code from the pitch contour feature. The rhythm code is the rhythm code, the content code is the content code, and the pitch code is the pitch code. The rhythm code, the content code, and the pitch code are all hidden representations (hidden vectors) or disentangled representations.

[0055] Because the timbre feature is equivalent to the user identity feature and is directly provided to the decoder, all encoders do not need to encode the timbre feature. Different encoders encode content, pitch and rhythm separately.

[0056] The rhythm encoder is the only one that can complete the rhythm encoding. Other encoders can only obtain partial rhythm features. And other encoders have no way to guess the missing part of the rhythm features and provide the missing part. Therefore, the rhythm encoder must pass all the rhythm features.

[0057] Since all features except rhythm feature exist in other encoders, the rhythm encoder will force the corresponding information bottleneck to discard content feature, pitch feature and timbre feature in priority order.

[0058] The rhythm encoder only encodes rhythm features, while the input of the pitch encoder is the pitch contour feature without content features, so only the content encoder can be used to encode all content features. The content encoder will use the corresponding information bottleneck to discard rhythm features, pitch features, and timbre features and only encode content features.

[0059] The input of the pitch encoder contains some rhythm features and all pitch features. Since the pitch encoder cannot guess the missing part of the rhythm features, the pitch encoder will use the corresponding information bottleneck to encode only the pitch features and discard other features.

[0060] S500: Decoding the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in the pre-trained speech feature decoupling model to obtain a second spectrum feature.

[0061] Specifically, the user identity feature is a feature vector. For example, the user identity feature can be encoded as a one-hot vector or embedding, which is provided to the decoder separately.

[0062] The decoder reconstructs the inputs of rhythm coding, content coding, pitch coding, and user identity features to generate a new speech spectrum, namely, the second spectrum feature, thereby reconstructing the input first spectrum feature.

[0063] S600: Calculate a reconstruction loss function according to the first spectrum feature and the second spectrum feature.

[0064] Specifically, the calculation formula of the reconstruction loss function is as shown in formula (1):

[0065]

[0066] Among them, L recon To reconstruct the loss function, θ is all the trainable parameters of the model, is the predicted second spectrum feature, and S is the first spectrum feature.

[0067] S700: Calculate an orthogonal loss function based on the correlation between audio component pairs, wherein each group of audio component pairs includes two different audio components, and the audio components include rhythm coding, pitch coding and content coding, or the audio components include rhythm coding, pitch coding, timbre coding and content coding.

[0068] Specifically, the audio component is a component feature of the audio. In this embodiment, there is a correlation between every two audio components, and the correlation is the vector distance between the two audio components. The orthogonal loss function is calculated based on the correlation between all audio component pairs, and each group of audio component pairs includes two different audio components.

[0069] If the audio components include rhythm coding, pitch coding and content coding, the orthogonal loss function is calculated based on the correlation between the rhythm coding and the pitch coding, the correlation between the rhythm coding and the content coding, and the correlation between the pitch coding and the content coding.

[0070] If the audio components include rhythm coding, pitch coding, timbre coding and content coding, the orthogonal loss function is calculated based on the correlation between the rhythm coding and the pitch coding, the correlation between the rhythm coding and the timbre coding, the correlation between the rhythm coding and the content coding, the correlation between the pitch coding and the timbre coding, the correlation between the pitch coding and the content coding, and the correlation between the timbre coding and the content coding.

[0071] The correlation or correlation relationship between the audio components can be represented by the orthogonality MI between the audio components or by the similarity between the audio components. The orthogonality MI can be represented by mutual information, but the present application is not limited thereto.

[0072] By minimizing the orthogonal loss function, the correlation between multiple audio components can be minimized, making the separated or decoupled audio components as unrelated or non-overlapping as possible.

[0073] S800: Obtaining an overall loss function of the pre-trained speech feature decoupling model according to the reconstruction loss function and the orthogonal loss function.

[0074] Specifically, the overall loss function can be the weighted sum of the reconstruction loss function and the orthogonal loss function, or the sum of the products of the reconstruction loss function and the orthogonal loss function with the weights.

[0075] S900: Calculate the gradient according to the overall loss function, and iteratively update the model parameters of the pre-trained speech feature decoupling model according to the gradient until a convergence condition is reached, thereby obtaining the trained speech feature decoupling model.

[0076] Specifically, the convergence condition may be that the number of training times reaches a threshold or that the overall loss function is reduced to a certain threshold. If the convergence condition is not reached, the gradient is calculated according to the overall loss function during the training of the model, and the parameters of the encoder or the encoder and decoder are iteratively updated according to the gradient to obtain an updated pre-trained speech feature decoupling model. Then select new sample audio and execute steps S100-S900 again to train the updated pre-trained speech feature decoupling model, and execute the cycle until the convergence condition is reached.

[0077] This embodiment constructs a speech feature decoupling model, and trains the rhythm encoder, pitch encoder, content encoder and decoder included in the speech feature decoupling model, calculates the overall loss function by reconstructing the loss function and the orthogonal loss function, and iteratively optimizes the model parameters according to the overall loss function for the purpose of minimizing the overall loss function as much as possible, so as to further optimize the decoupling ability of the speech feature decoupling model, so that the speech feature decoupling model obtained after training can fully decouple the speech, and each component feature is independently decomposed. In addition, the training samples of this embodiment do not require complex sample labels, and the sample labels are easy to obtain.

[0078] In one embodiment, step S500 specifically includes:

[0079] Up-sampling the rhythm code, the content code, and the pitch code to obtain a first sampling feature;

[0080] Repeat the user identity feature along the time dimension that matches the first sampling feature to obtain a second sampling feature;

[0081] The first sampling feature and the second sampling feature are concatenated and predicted along the channel dimension to obtain a second spectrum feature.

[0082] Specifically, because the timbre feature is equivalent to the user identity feature and is directly provided to the decoder, all encoders do not need to encode the timbre feature.

[0083] The decoder includes an upsampling layer, a connection layer, a BLSTM layer and a Linear layer. The upsampling layer upsamples the rhythm code, content code and pitch code for the hidden representation to obtain the first sampling feature for the upsampling representation to restore the original sampling rate.

[0084] The user identity label is a one-hot vector, which is also repeated along the time dimension to match the time dimension of other upsampled representations. All upsampled representations consisting of the first sampled feature and the second sampled feature are concatenated along the channel dimension and given to the BLSTM layer of the decoder. The output of the BLSTM layer of the decoder is given to the linear layer to obtain the final output, which is the second spectral feature.

[0085] In one embodiment, step S700 specifically includes:

[0086] Calculate the mutual information between each two audio components;

[0087] The upper limit of the obtained mutual information is taken and summed to obtain the orthogonal loss function.

[0088] Specifically, the correlation between audio components in this embodiment is represented by orthogonality MI, which is represented by mutual information. Given random variables u and v, orthogonality MI is the Kullback-Leibler (KL) divergence between their joint distribution and marginal distribution, and mutual information is used to represent the divergence.

[0089] Among them, the mutual information is I(u,v)=D KL (P(u,v); P(u)P(v)) Formula (2)

[0090] Among them, P(u,v), P(u), and P(v) are all probability distributions.

[0091] The upper limit of MI orthogonality is:

[0092]

[0093] The audio components of this embodiment include content coding, timbre coding, pitch coding and rhythm coding. s, p, and r are content coding, timbre coding, pitch coding, and rhythm coding, respectively. is a variational approximation of the true posterior of u given v, and can be obtained through the model parameters Parameterize. E is the expected function.

[0094] In one embodiment, the calculation formula of the overall loss is shown in formula (4):

[0095] L=L reco n+λL MI Formula (4)

[0096] Among them, L is the overall loss function, L recon is the self-reconstruction loss function, L MI is the orthogonal loss function, λ is the loss weight of the orthogonal loss function to the overall loss function;

[0097] The calculation formula of the orthogonal loss function is shown in formula (5):

[0098]

[0099] Among them, u and v are the codes corresponding to two randomly given audio components. is the upper limit of the mutual information. The loss weight λ is a given constant, which represents the influence of the orthogonal loss function on the overall loss function.

[0100] This embodiment increases the degree of decoupling between the audio features by reducing the mutual information between the audio features.

[0101] In one embodiment, step S300 specifically includes:

[0102] Performing a first convolution process on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model,

[0103] Performing a first random resampling operation on the output obtained by the first convolution process to obtain a third sampling feature,

[0104] The third sampling feature is processed by the first BLSTM layer to obtain the first intermediate feature.

[0105] Down-sampling the first intermediate feature to obtain content encoding;

[0106] Step S400 specifically includes:

[0107] The pitch contour feature is processed by the second convolution using the pitch encoder included in the pre-trained speech feature decoupling model.

[0108] A second random resampling operation is performed on the output obtained by the second convolution process to obtain a fourth sampling feature,

[0109] The fourth sampling feature is processed by the second BLSTM layer to obtain the second intermediate feature.

[0110] The second intermediate feature is downsampled to obtain pitch encoding.

[0111] Specifically, a first random resampling operation is performed on the output obtained by the first convolution process to obtain a first sampling feature, including: dividing the output obtained by the first convolution process into multiple first segments of random lengths, and randomly stretching or compressing each first segment along the time dimension to obtain a corresponding first sampling feature.

[0112] After the first spectrum feature is randomly resampled, the rhythm feature will randomly lose some features, while other features remain unchanged.

[0113] Performing a second random resampling operation on the output obtained by the second convolution processing to obtain a second sampling feature, including: dividing the output obtained by the second convolution processing into a plurality of second segments of random lengths, and randomly stretching or compressing each second segment along the time dimension to obtain a corresponding second sampling feature.

[0114] The pitch contour itself contains incomplete rhythmic features, and the pitch contour features will lose more rhythmic features after random stretching or compression in random resampling, so it cannot be encoded using the destroyed or distorted rhythmic features.

[0115] The network structures of the rhythm encoder, content encoder and pitch encoder are similar. The content encoder and pitch encoder both include a set of 5x 1 convolutional layers, group normalization layers or group normalization layers, random resampling layers, BLSTM layers and downsampling layers. The rhythm encoder includes a set of 5x 1 convolutional layers, group normalization layers or group normalization layers, BLSTM layers and downsampling layers.

[0116] The outputs of the convolutional layers of the content encoder and the pitch encoder are given to the random resampling layer to further contaminate or disrupt the rhythm features. The output obtained by the sampling of the convolutional layer is given to the BLSTM layer of the encoder to reduce the feature size or reduce the feature dimension; the output of the BLSTM layer of the encoder is downsampled to reduce the time dimension, thereby generating a hidden representation, that is, the corresponding content encoding or pitch encoding.

[0117] Since each encoder adds a corresponding information bottleneck, each encoder does not encode timbre features.

[0118] This embodiment ensures that each encoder encodes the component features that it is responsible for, so as to achieve the purpose of feature decoupling.

[0119] In one embodiment, the method further comprises:

[0120] Corresponding information bottlenecks are added to the rhythm encoder, content encoder, and pitch encoder respectively.

[0121] The principle of speech feature decoupling in this application is described in detail below:

[0122] Assuming that speech data can be decoupled into four components: rhythm, pitch, content, and timbre, we proposed this model in the hope of completely decoupling the four speech components and achieving individual control of each component.

[0123] Timbre features: User identity features provide timbre information, so it can be assumed that other encoders no longer need to encode timbre features.

[0124] Rhythm features: Since the regional spectral features of the input content encoder and the pitch contour features of the input pitch encoder have been randomly resampled, the speech rhythm is disrupted. Therefore, the rhythm information input to the content encoder and the pitch encoder is incomplete. Therefore, the complete rhythm information can only be encoded by the rhythm encoder. The information bottleneck added to the rhythm encoder is used to make the rhythm encoder only encode the rhythm features and discard other features. In this way, the rhythm encoder will only output the rhythm code.

[0125] Pitch feature: Since the input to the pitch encoder is the normalized pitch contour feature, and the pitch contour feature only contains the pitch feature and incomplete rhythm feature. And the incomplete rhythm feature will further confirm some rhythm features after random resampling. When the rhythm feature is provided by the rhythm encoder, the information bottleneck added to the pitch encoder makes the feature encoded by the pitch encoder only the pitch feature, that is, the output is only the pitch code.

[0126] Content features: Since the other three information of speech information: rhythm and pitch have been provided by the other two encoders, and the first spectral feature will be randomly resampled after being input into the content encoder, its complete rhythm feature will be randomly discarded. The information bottleneck added to the content encoder is used to make the content encoder discard other features and only encode the content features, and the expected output is content encoding.

[0127] This application uses an encoder-decoder structure. The encoder limits the output dimension and the added information bottleneck filters out other invalid features. Random sampling in the time dimension further filters out disruptive rhythm features, and different codes are passed to the decoder through different encoders, thereby decoupling the four parts of timbre, pitch, rhythm, and content from the speech.

[0128] The present application also provides a method for decoupling speech features, the method comprising:

[0129] Obtaining target spectral features of the speech to be decoupled and target pitch contour features obtained after normalization;

[0130] The target spectral features are input into the rhythm encoder and content encoder of the trained speech feature decoupling model obtained according to any of the previous items, and the target pitch contour features are input into the pitch encoder of the trained speech feature decoupling model for encoding and decoupling respectively to obtain the target rhythm code, target content code and target pitch code.

[0131] Specifically, the trained speech feature decoupling model includes an encoder and a decoder, and the decoder is used to synthesize or reconstruct the spectral features corresponding to the speech. Therefore, if it is necessary to extract the audio components or component features in the audio or speech, only the encoder of the trained speech feature decoupling model needs to be used. Through each trained encoder, the audio components in the decoupled speech can be disentangled to obtain the corresponding target rhythm code, target content code and target pitch code.

[0132] The present application also provides a speech synthesis method, the method comprising:

[0133] Obtaining a third spectrum feature and a normalized second pitch contour feature of each candidate speech;

[0134] Inputting one of the third spectral features into a rhythm encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target rhythm encoding;

[0135] Inputting one of the third spectral features into a content encoder of the trained speech feature decoupling model for encoding to obtain a target content encoding;

[0136] Inputting one of the second pitch contour features into a pitch encoder of a trained speech feature decoupling model for encoding to obtain a target pitch encoding;

[0137] Inputting the target user identity feature selected from the first user identity feature corresponding to the candidate voice and the obtained target rhythm code, target content code, and target pitch code into the decoder of the trained voice feature decoupling model for decoding to obtain a synthesized target spectrum feature, wherein the inputs of the rhythm encoder, content encoder, and pitch encoder in the trained voice feature decoupling model and the target user identity features input into the decoder of the trained voice feature decoupling model are not all features corresponding to the same candidate voice;

[0138] The target spectral features are converted into target synthesized speech using a vocoder.

[0139] Specifically, speech synthesis, as an important branch of artificial intelligence, has been applied to all aspects of our daily lives. It synthesizes natural audio like human speech by inputting text through artificial intelligence algorithms. The speech synthesis model of this embodiment includes an acoustic model and a vocoder, wherein the acoustic model includes a trained speech feature decoupling model obtained by any of the above items.

[0140] Take speech synthesis of two candidate voices as an example. The speech synthesis method is specifically as follows:

[0141] Acquire a third spectrum feature of the first speech and a second pitch contour feature obtained after normalization;

[0142] Acquire a fourth frequency spectrum feature of the second speech and a third pitch contour feature obtained after normalization;

[0143] Inputting one of the third spectral feature and the fourth spectral feature into a rhythm encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target rhythm encoding;

[0144] Inputting one of the third spectral feature and the fourth spectral feature into a content encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target content encoding;

[0145] Inputting one of the second pitch contour feature and the third pitch contour feature into a pitch encoder of the trained speech feature decoupling model obtained according to any of the preceding items for encoding to obtain a target pitch encoding;

[0146] Inputting a target user identity feature selected from a first user identity feature corresponding to a first voice and a second user identity feature corresponding to a second voice, as well as a target rhythm code, a target content code, and a target pitch code obtained, into a decoder of a trained voice feature decoupling model obtained according to any of the preceding items for decoding to obtain a synthesized target spectral feature, wherein the inputs of a rhythm encoder, a content encoder, and a pitch encoder in the trained voice feature decoupling model and the target user identity features input into the decoder of the trained voice feature decoupling model are not all features corresponding to the same voice;

[0147] The target spectral features are converted into target synthesized speech using a vocoder.

[0148] Input one of the third spectrum feature and the fourth spectrum feature into the rhythm encoder and the content encoder of the trained speech feature decoupling model. The inputs of the rhythm encoder and the content encoder may be the same or different. Input one of the second pitch contour feature and the third pitch contour feature into the pitch encoder of the trained speech feature decoupling model. Input one of the first user identity feature corresponding to the first speech and the second user identity feature corresponding to the second speech into the decoder of the trained speech feature decoupling model. It is necessary to ensure that the inputs of the encoder and the decoder of the trained speech feature decoupling model are not all from the features corresponding to the same speech, so as to ensure that the features input to the decoder are from the features of two different speech, so as to achieve the purpose and meaning of speech synthesis.

[0149] This embodiment uses at least two candidate voices, and may use three or four candidate voices without being limited thereto.

[0150] The function of the vocoder is to convert the obtained Mel graph into speech. This embodiment realizes speech synthesis of various component features from the perspective of timbre, pitch, content and rhythm, enriching the application of speech synthesis.

[0151] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0152] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0153] Figure 3 This is a structural block diagram of a training device for a speech feature decoupling model in an embodiment of the present application, refer to Figure 3 , the device comprises:

[0154] The data acquisition module 100 is used to acquire the first spectrum feature of the sample audio and the pitch contour feature obtained after standardization;

[0155] A rhythm encoding module 200, used for inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code;

[0156] The content encoding module 300 is used to perform a second encoding on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model to obtain a content code;

[0157] A pitch coding module 400 is used to perform a third coding on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch code;

[0158] A decoding module 500 is used to decode the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in a pre-trained speech feature decoupling model to obtain a second spectrum feature;

[0159] A first calculation module 600, configured to calculate a reconstruction loss function according to the first spectrum feature and the second spectrum feature;

[0160] A second calculation module 700 is used to calculate an orthogonal loss function according to the correlation between every two audio components, wherein the audio components include rhythm coding, pitch coding and content coding, or the audio components include rhythm coding, pitch coding, timbre coding and content coding;

[0161] A third calculation module 800 is used to obtain an overall loss function of the pre-trained speech feature decoupling model according to the reconstruction loss function and the orthogonal loss function;

[0162] The parameter updating module 900 is used to calculate the gradient according to the overall loss function, and iteratively update the model parameters of the pre-trained speech feature decoupling model according to the gradient until the convergence condition is reached, thereby obtaining the trained speech feature decoupling model.

[0163] In one embodiment, the decoding module 500 includes:

[0164] A first upsampling module, used for upsampling the rhythm code, the content code, and the pitch code to obtain a first sampling feature;

[0165] A second upsampling module, configured to repeat the user identity feature along a time dimension matching the first sampling feature to obtain a second sampling feature;

[0166] The synthesis module is used to perform splicing prediction on the first sampling feature and the second sampling feature along the channel dimension to obtain a second spectrum feature.

[0167] In one embodiment, the second calculation module 700 includes:

[0168] A mutual information calculation module, used for calculating the mutual information between every two audio components;

[0169] The summation module is used to take the upper limit of the obtained mutual information and then sum it to obtain the orthogonal loss function.

[0170] In one embodiment, the overall loss is calculated as:

[0171] L=L recon +λL MI

[0172] Among them, L is the overall loss function, L recon is the self-reconstruction loss function, L MI is the orthogonal loss function, λ is the loss weight of the orthogonal loss function to the overall loss function;

[0173] The calculation formula of the orthogonal loss function is:

[0174]

[0175] Among them, u and v are the codes corresponding to two randomly given audio components. is the upper limit of mutual information.

[0176] In one embodiment, the content encoding module 300 includes:

[0177] A first convolution module is used to perform a first convolution process on the first spectrum feature using a content encoder included in the pre-trained speech feature decoupling model,

[0178] A first random resampling module is used to perform a first random resampling operation on the output obtained by the first convolution process to obtain a third sampling feature.

[0179] The first BLSTM module is used to process the third sampling feature using the first BLSTM layer to obtain a first intermediate feature.

[0180] A first downsampling module, used to downsample the first intermediate feature to obtain a content code;

[0181] The pitch encoding module 400 includes:

[0182] The second convolution module is used to perform a second convolution process on the pitch contour feature using the pitch encoder included in the pre-trained speech feature decoupling model.

[0183] A second random resampling module is used to perform a second random resampling operation on the output obtained by the second convolution process to obtain a fourth sampling feature.

[0184] The second BLSTM module is used to process the fourth sampling feature using the second BLSTM layer to obtain a second intermediate feature.

[0185] The second downsampling module is used to downsample the second intermediate feature to obtain pitch encoding.

[0186] In one embodiment, the apparatus further comprises:

[0187] The bottleneck information adding module is used to add corresponding information bottlenecks to the rhythm encoder, content encoder and pitch encoder respectively.

[0188] The present application also provides a speech feature decoupling device, the device comprising:

[0189] A target data acquisition module is used to obtain the target spectrum characteristics of the speech to be decoupled and the target pitch contour characteristics obtained after normalization;

[0190] The feature decoupling module is used to input the target spectral features into the rhythm encoder and content encoder of the trained speech feature decoupling model obtained in any of the previous items, and input the target pitch contour features into the pitch encoder of the trained speech feature decoupling model for encoding and decoupling respectively, so as to obtain the target rhythm code, target content code and target pitch code.

[0191] The present application also provides a speech synthesis device, which comprises:

[0192] A first data acquisition module, used for acquiring a third spectrum feature of the first speech and a second pitch contour feature obtained after normalization;

[0193] A second data acquisition module, used for acquiring a fourth frequency spectrum feature of the second speech and a third pitch contour feature obtained after normalization;

[0194] A target rhythm encoding module, used for inputting one of the third spectrum feature and the fourth spectrum feature into a rhythm encoder of the trained speech feature decoupling model obtained in any of the above items for encoding to obtain a target rhythm encoding;

[0195] A target content encoding module, used for inputting one of the third spectrum feature and the fourth spectrum feature into a content encoder of the trained speech feature decoupling model for encoding to obtain a target content encoding;

[0196] A target pitch encoding module, used for inputting one of the second pitch contour feature and the third pitch contour feature into a pitch encoder of a trained speech feature decoupling model for encoding to obtain a target pitch encoding;

[0197] A target decoding module, for inputting a target user identity feature selected from a first user identity feature corresponding to a first voice and a second user identity feature corresponding to a second voice, and the obtained target rhythm code, target content code, and target pitch code into a decoder of a trained voice feature decoupling model for decoding to obtain a synthesized target spectral feature, wherein the inputs of the rhythm encoder, content encoder, and pitch encoder in the trained voice feature decoupling model and the target user identity feature input into the decoder of the trained voice feature decoupling model are not features corresponding to the same voice;

[0198] The conversion module is used to convert the target spectrum features into the target synthesized speech by using a vocoder.

[0199] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0200] The meaning of "first" and "second" in the above modules / units is only to distinguish different modules / units, and is not used to define which module / unit has a higher priority or other limiting meanings. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in this application is only a logical division, and there may be other division methods when implemented in actual applications.

[0201] For the specific definition of the training device of the speech feature decoupling model, please refer to the definition of the training method of the speech feature decoupling model in the above text, which will not be repeated here. The various modules in the training device of the above speech feature decoupling model can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0202] Figure 4 FIG. 1 is a block diagram of the internal structure of a computer device in one embodiment of the present application. Figure 4 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory includes a storage medium and an internal memory. The storage medium can be a non-volatile storage medium or a volatile storage medium. The storage medium stores an operating system and can also store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can implement a training method for a speech feature decoupling model, or a speech feature decoupling method, or a speech synthesis method. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the storage medium. The internal memory can also store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a training method for a speech feature decoupling model, or a speech feature decoupling method, or a speech synthesis method. The network interface of the computer device is used to communicate with an external server through a network connection. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0203] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions (e.g., a computer program) stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the method for training the speech feature decoupling model in the above embodiment (e.g., Figure 1 Steps S100 to S900 shown in the figure and other extensions of the method and extensions of related steps), or, a method for decoupling speech features, or, a method for speech synthesis. Alternatively, when the processor executes the computer-readable instructions, the functions of each module / unit of the training device for the speech feature decoupling model in the above-mentioned embodiment are realized, for example Figure 3 The functions of the modules 100 to 900 are shown in detail and will not be described again to avoid repetition.

[0204] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0205] The memory can be used to store computer-readable instructions and / or modules. The processor implements various functions of the computer device by running or executing the computer-readable instructions and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.), etc.

[0206] The memory may be integrated into the processor or provided separately from the processor.

[0207] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0208] In one embodiment, a computer-readable storage medium is provided on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the training method of the speech feature decoupling model in the above embodiment are implemented (for example, Figure 1 Steps S100 to S900 shown in the figure and other extensions of the method and extensions of related steps), or, a method for decoupling speech features, or, a method for speech synthesis. Alternatively, when the computer-readable instructions are executed by a processor, the functions of each module / unit of the training device for the speech feature decoupling model in the above-mentioned embodiment are realized, for example Figure 3 The functions of the modules 100 to 900 are shown in detail and will not be described again to avoid repetition.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0210] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0211] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. Through the description of the above implementation modes, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation mode. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions to enable a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0212] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A training method for a speech feature decoupling model, characterized in that: The method comprises: Obtaining a first spectrum feature of the sample audio and a pitch contour feature obtained after normalization; Inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code; Performing a second encoding on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model to obtain a content code; Performing a third encoding on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch code; Decoding the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in the pre-trained speech feature decoupling model to obtain a second spectrum feature; Calculating a reconstruction loss function according to the first spectrum feature and the second spectrum feature; Calculating an orthogonal loss function according to correlations between pairs of audio components, wherein each group of audio component pairs includes two different audio components, and the audio components include rhythm coding, pitch coding, and content coding, or the audio components include rhythm coding, pitch coding, timbre coding, and content coding; Obtaining an overall loss function of the pre-trained speech feature decoupling model according to the reconstruction loss function and the orthogonal loss function; The gradient is calculated according to the overall loss function, and the model parameters of the pre-trained speech feature decoupling model are iteratively updated according to the gradient until a convergence condition is reached, thereby obtaining a trained speech feature decoupling model.

2. The method according to claim 1, characterized in that The decoder included in the pre-trained speech feature decoupling model decodes the rhythm code, content code, pitch code, and user identity feature input into the decoder to obtain a second spectrum feature, including: Upsampling the rhythm code, the content code, and the pitch code to obtain a first sampling feature; Repeating the user identity feature along the time dimension matching the first sampling feature to obtain a second sampling feature; The first sampling feature and the second sampling feature are concatenated and predicted along the channel dimension to obtain a second spectrum feature.

3. The method according to claim 1, characterized in that The calculating of the orthogonal loss function according to the correlation between the audio component pairs includes: Calculate the mutual information between each two audio components; The upper limit of the obtained mutual information is taken and summed to obtain the orthogonal loss function.

4. The method according to claim 3, characterized in that The calculation formula of the overall loss is: L=L recon +λL MI Among them, L is the overall loss function, L recon is the self-reconstruction loss function, L MI is the orthogonal loss function, λ is the loss weight of the orthogonal loss function to the overall loss function; The calculation formula of the orthogonal loss function is: Among them, u and v are the codes corresponding to two randomly given audio components. is the upper limit of the mutual information of each two audio components, They are content coding, timbre coding, pitch coding and rhythm coding.

5. The method according to claim 1, characterized in that The step of performing a second encoding on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model to obtain a content encoding includes: performing a first convolution process on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model, Performing a first random resampling operation on the output obtained by the first convolution process to obtain a third sampling feature, The third sampling feature is processed by the first BLSTM layer to obtain a first intermediate feature, Downsampling the first intermediate feature to obtain a content code; The pitch contour feature is thirdly encoded using the pitch encoder included in the pre-trained speech feature decoupling model to obtain the pitch encoding, including: Performing a second convolution process on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model, A second random resampling operation is performed on the output obtained by the second convolution process to obtain a fourth sampling feature, The fourth sampling feature is processed by the second BLSTM layer to obtain a second intermediate feature, The second intermediate feature is downsampled to obtain pitch encoding.

6. The method according to claim 1, characterized in that Before the decoder included in the pre-trained speech feature decoupling model decodes the rhythm code, content code, pitch code, and user identity feature input into the decoder to obtain the second spectrum feature, the method further includes: Corresponding information bottlenecks are added to the rhythm encoder, content encoder and pitch encoder respectively.

7. A method for decoupling speech features, characterized in that: The method comprises: Obtaining target spectral features of the speech to be decoupled and target pitch contour features obtained after normalization; The target spectral features are input into the rhythm encoder and content encoder of the trained speech feature decoupling model obtained according to any one of claims 1-6, and the target pitch contour features are input into the pitch encoder of the trained speech feature decoupling model for encoding and decoupling respectively to obtain target rhythm coding, target content coding and target pitch coding.

8. A speech synthesis method, characterized in that: The method comprises: Obtaining a third spectrum feature and a normalized second pitch contour feature of each candidate speech; Inputting one of the third spectral features into a rhythm encoder of a trained speech feature decoupling model obtained according to any one of claims 1 to 6 for encoding to obtain a target rhythm encoding; Inputting one of the third spectral features into the content encoder of the trained speech feature decoupling model for encoding to obtain a target content encoding; Inputting one of the second pitch contour features into the pitch encoder of the trained speech feature decoupling model for encoding to obtain a target pitch encoding; Inputting the target user identity feature selected from the first user identity feature corresponding to the candidate voice and the obtained target rhythm code, target content code, and target pitch code into the decoder of the trained voice feature decoupling model for decoding to obtain a synthesized target spectrum feature, wherein the inputs of the rhythm encoder, content encoder, and pitch encoder in the trained voice feature decoupling model and the target user identity features input into the decoder of the trained voice feature decoupling model are not all features corresponding to the same candidate voice; The target spectral features are converted into target synthesized speech using a vocoder.

9. A training device for a speech feature decoupling model, characterized in that: The device comprises: A data acquisition module, used to acquire a first frequency spectrum feature of the sample audio and a pitch contour feature obtained after standardization; A rhythm encoding module, used for inputting the first spectrum feature into a rhythm encoder included in a pre-trained speech feature decoupling model for first encoding to obtain a rhythm code; A content encoding module, configured to perform a second encoding on the first spectral feature using a content encoder included in the pre-trained speech feature decoupling model to obtain a content code; A pitch coding module, used for performing a third coding on the pitch contour feature using a pitch encoder included in the pre-trained speech feature decoupling model to obtain a pitch code; A decoding module, configured to decode the rhythm code, content code, pitch code, and user identity feature input into the decoder through a decoder included in the pre-trained speech feature decoupling model to obtain a second spectrum feature; A first calculation module, used for calculating a reconstruction loss function according to the first spectrum feature and the second spectrum feature; A second calculation module is used to calculate an orthogonal loss function according to the correlation between the audio component pairs, wherein each group of the audio component pairs includes two different audio components, and the audio components include rhythm coding, pitch coding and content coding, or the audio components include rhythm coding, pitch coding, timbre coding and content coding; A third calculation module, used for obtaining an overall loss function of the pre-trained speech feature decoupling model according to the reconstruction loss function and the orthogonal loss function; A parameter updating module is used to calculate the gradient according to the overall loss function, and iteratively update the model parameters of the pre-trained speech feature decoupling model according to the gradient until the convergence condition is reached, thereby obtaining the trained speech feature decoupling model.

10. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, it executes the steps of the training method of the speech feature decoupling model as described in any one of claims 1 to 6, or, when the processor executes the computer-readable instructions, it executes the steps of the speech feature decoupling method as described in claim 7, or, when the processor executes the computer-readable instructions, it executes the steps of the speech synthesis method as described in claim 8.

11. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the processor executes the steps of the training method of the speech feature decoupling model as described in any one of claims 1 to 6, or the processor executes the steps of the speech feature decoupling method as described in claim 7, or the processor executes the steps of the speech synthesis method as described in claim 8.

Images