Method for training a pitch-based voice conversion model and voice conversion system

Abstract

A method and system for training a pitch-based voice conversion model are provided. [Solution] The training method includes the steps of outputting an audio feature code by a pre-encoder, extracting pitch features by a pitch extraction module, inputting a linear spectrum corresponding to the reference voice into a post-encoder to obtain audio hidden variables, stitching the audio feature code and the pitch features to obtain an audio stitching feature, inputting the obtained audio stitching feature and the audio hidden variables into a timing alignment module to obtain a converted voice code, decoding the converted voice code by a decoder to obtain a converted voice, and calculating the training loss of the converted voice to determine the convergence degree of the voice conversion model.

Description

[Technical field]

[0001] The present application relates to the technical field of voice conversion, and in particular to a method for training a pitch-based voice conversion model and a voice conversion system. [Background technology]

[0002] Voice conversion technology is a technology that converts one person's voice into another person's voice using a voice conversion model. In order to obtain a high-quality human voice, however, it is necessary to train the voice conversion model using a large amount of sample data. In this way, a more realistic human voice can be obtained through voice conversion technology.

[0003] In the actual training process, it is often difficult to obtain a large amount of sample data, so the voice conversion model can only be trained with a small amount of sample data. However, because the natural human voice has rich expressiveness, the human voice has large changes in timbre and rhythm, while the human voice generated by the voice conversion model trained with a small amount of sample data has a certain difference from the real human voice.

[0004] When training a voice conversion model using a small amount of sample data to reduce the difference in voice, a pre-training and model fine-tuning method can be adopted, that is, a voice conversion model is first pre-trained on a large amount of audio data data set, and then the voice conversion model is fine-tuned using a small amount of sample data. However, when using a small amount of sample data, there is still a problem that the pitch similarity between the converted person's voice and the real person's voice is low. Summary of the Invention [Problem to be solved by the invention]

[0005] This is to reduce the problem of low pitch similarity between the converted human voice and the real human voice when the amount of sample data is small. [Means for solving the problem]

[0006] In a first aspect, some embodiments of the present application provide a pitch-based voice conversion model training method applied to training a voice conversion model, the voice conversion model including a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module, the method including: Inputting a reference speech into the pre-encoder and the pitch extraction module, extracting audio feature codes by the pre-encoder and extracting pitch features by the pitch extraction module; performing feature stitching on the audio feature code and the pitch feature to obtain an audio stitching feature; inputting a linear spectrum corresponding to the reference speech into the post-encoder to obtain audio hidden variables; Aligning the timing sequence of the audio stitching feature and the audio latent variable by the timing alignment module to obtain a converted audio code; decoding the converted speech code by the decoder to obtain a converted speech; calculating a training loss of the converted speech, and if the training loss is less than or equal to a training loss threshold, outputting a speech conversion model based on current parameters of a trained model, and if the training loss is greater than the training loss threshold, performing iterative training on the speech conversion model, wherein the trained model is a speech conversion model that has not been trained to convergence.

[0007] In some embodiments, the pitch extraction module includes an encoder layer, a filter layer, an intermediate layer, and a decoder layer; The encoder layer, the hidden layer and the decoder layer form a first encoding branch of the pitch extraction module, and the encoder layer, the filter layer and the decoder layer form a second encoding branch of the pitch extraction module.

[0008] In some embodiments, the encoder layer includes an average pooling layer and a convolutional network, and the step of extracting pitch features by the pitch extraction module includes: extracting a pitch feature vector of the reference voice by the convolutional network; performing downsampling on the pitch feature vector by the average pooling layer to obtain a pitch feature code; performing decoding on the pitch feature code by the decoder layer to obtain the pitch feature.

[0009] In some embodiments, the convolutional network includes a convolutional block, the convolutional block including a 2D convolutional layer, a batch normalization layer and a relu function, and the step of extracting the pitch feature vector of the reference voice by the convolutional network includes: Extracting deep audio vectors using the 2D convolutional layer; performing a fast convergence process on the deep audio vectors by the batch normalization layer to extract convergent pitch features from the deep audio vectors; adding a non-linear relationship to the converged pitch feature by the relu function to obtain the pitch feature vector.

[0010] In some embodiments, a shortcut convolution layer is provided between the input end of the convolution network and the output end of the convolution network, and before the step of adding a nonlinear relationship to the convergent pitch feature by the relu function, extracting shortcut pitch features by the shortcut convolutional layer; stitching the shortcut pitch features and the convergent pitch features to obtain a pitch stitching feature; and adding a nonlinear relationship between the pitch stitching features by the relu function to obtain the pitch feature vector.

[0011] In some embodiments, the decoder layer includes a deconvolutional layer and the convolutional network, and the step of performing a decoding operation on the pitch feature code by the decoder layer includes: performing a deconvolution calculation on the pitch feature code by the deconvolution layer to obtain a deconvolution feature vector; performing decoding on the deconvolved feature vector by the convolutional network to obtain the pitch feature.

[0012] In some embodiments, the step of aligning timing sequences of the audio stitching features and the audio latent variables by the timing alignment module comprises: obtaining a template speech sequence for a timing alignment module; aligning timing sequences of the audio stitching features and the audio latent variables based on the template audio sequence; and performing encoding on the aligned speech stitching features and the audio latent variables to obtain a transformed speech code.

[0013] In some embodiments, the voice conversion model further comprises a style encoder, and after the step of performing feature stitching on the audio feature codes and the pitch features, the method further comprises: extracting style features of the reference audio by the style encoder; and updating the audio stitching features by mapping the style features to the audio stitching features.

[0014] In some embodiments, the training loss comprises a spectral loss, and the step of calculating the training loss for the converted speech comprises: obtaining a spectral accuracy of the reference speech and obtaining a spectral accuracy of a converted speech; Calculating the spectral loss based on the spectral accuracy of the reference speech and the spectral accuracy of the converted speech according to the following formula:

number

[0015] In a second aspect, the present application provides a voice conversion system, comprising: a voice conversion model, the voice conversion model being obtained by training according to the pitch-based voice conversion model training method described in the first aspect, the voice conversion model including a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module, wherein the pre-encoder is configured to extract audio feature codes of a reference voice; the pitch extraction module is configured to extract pitch features of the reference voice; The post-encoder is configured to generate audio hidden variables based on a linear spectrum corresponding to the reference audio; The timing alignment module is configured to align a timing sequence of the audio stitching feature and the audio latent variable to obtain a converted audio code, and the audio stitching feature is obtained by combining the audio feature code and the pitch feature; The decoder is configured to perform decoding on the converted audio code to obtain a converted audio. Effect of the Invention

[0016] As can be seen from the above technical solutions, the present application provides a pitch-based voice conversion model training method and a voice conversion system, the method being used to train a voice conversion model, the voice conversion model including a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module. The method includes inputting a reference voice into a pre-encoder and a pitch extraction module, so that the pre-encoder outputs an audio feature code, and the pitch extraction module extracts a pitch feature. Then, the linear spectrum corresponding to the reference voice is input into the post-encoder to obtain an audio hidden variable. The audio feature code and the pitch feature are stitched to obtain an audio stitching feature and an audio hidden variable, which are input into a timing alignment module to obtain a converted voice code, and the converted voice code is decoded by a decoder to obtain a converted voice. The training loss of the converted voice is then calculated to determine the convergence degree of the voice conversion model. The present application uses a pitch extraction module to extract pitch features of the reference voice, stitches them with audio feature codes, and performs alignment so that the pitch features of the converted voice are more consistent with the real voice of a person, and improves the pitch similarity of the converted voice when voice samples are insufficient. [Brief description of the drawings]

[0017] In order to more clearly explain the technical solutions of the present application, the following briefly introduces the drawings that need to be used in the embodiments, and obviously, those skilled in the art can further derive other drawings based on these drawings without any creative efforts. [Figure 1] 1 is a flowchart of a method for training a pitch-based voice conversion model provided in an embodiment of the present application; [Diagram 2] FIG. 2 is a structural schematic diagram of a pitch extraction module in an embodiment of the present application. [Diagram 3] FIG. 1 is a structural schematic diagram of the REB network structure in an embodiment of the present application. [Figure 4] FIG. 2 is a structural schematic diagram of an RCB convolutional network structure in an embodiment of the present application. [Diagram 5] FIG. 2 is a structural schematic diagram of a network structure of a decoder layer in an embodiment of the present application; [Figure 6] FIG. 1 is a schematic flow chart of outputting a voice conversion model based on training loss in an embodiment of the present application; [Figure 7] FIG. 2 is a structural schematic diagram of a rhythm encoder in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present application will be described in detail below in combination with the embodiments with reference to the drawings. It should be noted that, unless inconsistent, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0019] In addition, the terms "first," "second," etc. in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or chronology.

[0020] Voice conversion technology is a technology that converts the voice of one person into the voice of another person through a voice conversion model. For example, when a user drives a digital human generated in his / her image, if the digital human wants to speak in a voice other than the user's own voice to enhance the interest of the interaction with the digital human, the user can use voice conversion technology to convert his / her own voice into the voice of another person, thereby achieving the purpose of voice conversion.

[0021] In order to improve the accuracy of voice conversion and obtain high-quality human-like voices, it is necessary to train the voice conversion model using a large amount of sample data, so that a more realistic human voice can be obtained through voice conversion technology.

[0022] However, in the actual training process, it is often difficult to obtain a large amount of sample data, so the voice conversion model can only be trained using a small amount of sample data. However, because the natural human voice has rich expressiveness, the voice of a person changes greatly in timbre and rhythm, while the converted voice generated by the voice conversion model obtained by training using a small amount of sample data has a certain difference from the actual voice of a person.

[0023] When the content of a language is fixed, the timbre and pitch of a speaker are factors that affect the speaker's personality. When the amount of speaker data is large, the voice conversion technology can produce good conversion results. However, when the amount of speaker data is small, the voice conversion technology has a low conversion effect in terms of the timbre and pitch of a speaker.

[0024] In some embodiments, when training a voice conversion model using a small amount of sample data, a pre-training and model fine-tuning method can be adopted to reduce the difference between the synthesized human voice and the real human voice, i.e., first pre-training the voice conversion model on a large amount of audio data dataset, and then fine-tuning the voice conversion model using a small amount of sample data. However, in the case of a small amount of sample data, the problem of low pitch similarity between the synthesized human voice and the real human voice still exists.

[0025] In order to reduce the problem of low pitch similarity between the converted human voice and a real human voice when a small amount of sample data is used, some embodiments of the present application provide a pitch-based voice conversion model training method, which is used to train a voice conversion model, where the use phase of the voice conversion model includes a pre-training phase, a learning training phase and an application phase.

[0026] In the pre-training stage of the voice conversion model, the voice conversion model can be pre-trained by training the voice, so that the voice conversion model has basic voice conversion ability. In the pre-training process, the training voice used can be obtained in various ways, for example, a corresponding audio file can be extracted from a video media resource as the training voice. In this example, if the duration of the audio file is long, a segmentation process can be performed on the audio file, thereby obtaining multiple training voice segments. Alternatively, some embodiments of the present application can perform voice synthesis on a preset text, thereby obtaining a training voice. The present application does not excessively restrict the acquisition method of the training voice.

[0027] In addition, since the training voice is used to perform a pre-training process for the voice conversion model, the training voice should include the voice of at least one person, so that voice conversion training is performed on the person's voice in the pre-training process of the voice conversion model.

[0028] The training method provided in the embodiment of the present application is used to train the learning training stage of a voice conversion model, in which the voice conversion model includes a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module, where the pre-encoder includes a normalizing flow layer, a projection layer and a content encoder.

[0029] In addition, in the embodiments of the present application, an audio feature code and a pitch feature can be added based on a speech synthesis algorithm to be used as a core algorithm of a speech conversion model. For example, a versatile and interpretable text-to-speech (VITS) algorithm, which synthesizes versatile and interpretable text into speech, can be used as a basic synthesis algorithm of a speech conversion model. Based on this algorithm, an audio encoder and an RMPVE pitch extraction algorithm can be added to be used as a core algorithm of a speech conversion model.

[0030] The VITS algorithm is a highly expressive speech synthesis algorithm that combines variational inference, normalized flow and adversarial training. The acoustic model and vocoder in speech synthesis are connected by hidden variables instead of spectra, and random modeling is performed on the hidden variables to use a random duration predictor to improve the diversity of the synthesized speech. The above is merely an exemplary description of the embodiment of the present application, and in actual applications, other speech synthesis algorithms can be used in combination with the speech conversion model of the embodiment of the present application.

[0031] 1 is a flowchart of a pitch-based voice conversion model training method provided in an embodiment of the present application. Referring to FIG. 1, the training method includes the following steps:

[0032] In S100, a reference speech is input to the pre-encoder and the pitch extraction module, and the pre-encoder extracts audio feature codes and the pitch extraction module extracts pitch features.

[0033] The reference voice is a voice to be trained on by the voice conversion model, for example, a user inputs the voice of person A into the voice conversion model, and the voice conversion model outputs the voice of person B based on the same text content as the voice of person A. Here, the voice of person B is the reference voice, and after the voice conversion model completes learning training on the reference voice, the target to be converted is input into the voice conversion model, and the voice conversion model performs voice conversion on the target voice with the voice features of the reference voice to obtain the converted voice.

[0034] It should be noted that the reference voice and the training voice are voice samples applied to two different stages of the voice conversion model, where the training voice is used to train the voice conversion model in the pre-training stage of the voice conversion model, so that the voice conversion model has basic voice conversion capabilities, and the reference voice is used to train the voice conversion model in the learning training stage of the voice conversion model, so that the voice conversion model can convert the voice features of the input voice into the voice features of the reference voice.

[0035] In this embodiment, in the learning and training process of the voice conversion model, the reference voice needs to be input to the audio encoder in the pre-encoder, so that the audio encoder extracts the audio feature code of the reference voice. In order to make the pitch of the converted voice output by the voice conversion model more consistent with the pitch of a real person's voice, the reference voice can be further synchronously input to the pitch extraction module, so that the pitch extraction module extracts the pitch feature of the reference voice, and improves the learning accuracy of the voice conversion model for the reference voice.

[0036] In some embodiments, the pre-encoder can further recognize the text content of the reference speech and extract features of the text information based on the text content, thereby improving the training accuracy of the speech conversion model to the reference speech.

[0037] In addition, in order to better extract audio features, a pre-processing can be performed on the reference speech before it is input to the audio encoder, for example, the reference speech is processed into a Mel spectrum, and the Mel spectrum is input to the audio encoder, so that the audio encoder extracts the Mel spectrum audio features and the text features corresponding to the reference speech, compresses and encodes the audio features and the text features, and outputs the audio feature code, so as to improve the accuracy of the speech obtained by subsequent conversion.

[0038] In some embodiments, as shown in Fig. 2, the pitch extraction module may include Encoder Layers, Filter Layers, Intermediate Layers and Decoder Layers, where the pitch extraction module may include two encoding branches: the Encoder Layers, the Intermediate Layers and the Decoder Layers form a first encoding branch of the pitch extraction module, and the Encoder Layers, the Filter Layers and the Decoder Layers form a second encoding branch of the pitch extraction module.

[0039] Here, the encoder layer is used to encode the reference speech, in the first encoding branch, an intermediate layer is used to extract high-level pitch features of the reference speech by multiple convolution layers, in the second encoding branch, a filter layer is used to skip hidden features in the reference speech and extract low-level pitch features in the reference speech by convolution layers, and the decoder layer is used to perform deconvolution operation on the high-level pitch features and low-level pitch features extracted by the first encoding branch and the second encoding branch, thereby completing decoding and obtaining pitch features.

[0040] In some embodiments, the encoder layer may include a preset number of REB network structures, where, as shown in FIG. 3 , the REB network structure includes multiple layers of RCB convolutional networks and one 2×2 average pooling layer (Avgpool), so that in the process of the pitch extraction module extracting the pitch feature of the reference voice, the pitch feature vector of the reference voice can be extracted by the RCB convolutional network.

[0041] The REB network structure includes multiple layers of RCB convolutional networks, and a 2x2 average pooling layer is placed after the last layer of the RCB convolutional network. The average pooling layer can perform downsampling on the pitch feature vector output by the RCB convolutional network to obtain a pitch feature code, and then the decoder layer can decode the pitch feature code to obtain pitch features.

[0042] Since the encoder layer includes multiple REB network structures, these REB network structures can sequentially increase the dimension for extracting pitch features according to the input order, and can extract pitch features from shallow to deep layers. For example, the extraction dimension of the REB network structure of the first layer is (1,16), the extraction dimension of the REB network structure of the second layer is (16,32), the extraction dimension of the REB network structure of the third layer is (32,64), and so on, thereby extracting pitch features step by step and stably, and improving the accuracy of pitch feature extraction.

[0043] In some embodiments, the RCB convolutional network may include a plurality of convolutional blocks, where the convolutional block may include a 2D convolutional layer, a batch normalization layer, and a relu function, as shown in FIG. 4, and the batch normalization layer and the relu function are placed after the 2D convolutional layer of each layer. In the process of extracting the pitch feature vector by the RCB convolutional network, the deep audio vector may be extracted by the 2D convolutional layer. In this embodiment, the accuracy of the audio vector may be determined by setting the number of layers of the 2D convolutional layer, and the more 2D convolutional layers are placed, the higher the accuracy of the audio vector should be. When the accuracy of the audio vector reaches its maximum, the number of layers of the 2D convolutional layer is the maximum number of layers, and the accuracy of the audio vector cannot be improved by increasing the 2D convolutional layer based on the maximum number of layers.

[0044] After the 2D convolutional layer extracts and obtains the audio vector, the audio vector needs to be input into the batch normalization layer of the corresponding convolutional block, and the batch normalization layer performs a fast convergence process on the deep audio vector to extract convergent pitch features from the deep audio vector, thereby improving the convergence speed of the RCB convolutional network and thereby improving the efficiency of voice conversion.

[0045] After obtaining the convergent pitch feature, a nonlinear relationship is added to the convergent pitch feature by the relu activation function, and a nonlinear characteristic is introduced into the convolution block to obtain a pitch feature vector, so that the pitch feature vector output by the current convolution block can be the input of the next convolution block, so that the pitch feature of the reference voice is extracted by multiple 2D convolution layers, and the pitch feature vector finally output by the convolution block is obtained. Note that, in order to improve the efficiency of extracting the pitch feature of the reference voice, in the embodiment of the present application, the optimal convolution core of the 2D convolution layer of the convolution block may be 3×3, and in actual application, the convolution core can be adjusted to 2×2 or 4×4 based on the efficiency required for the application, and the present application does not specifically limit the size of the convolution core.

[0046] In some embodiments, a shortcut convolution layer is further installed between the input end and the output end of the RCB convolution network, so that the shortcut pitch feature can be extracted by the shortcut convolution layer before the relu activation function adds a nonlinear relationship to the convergent pitch feature, where the shortcut convolution layer only includes one 2D convolution layer. Note that, since the speed at which the shortcut convolution layer extracts the shortcut pitch feature is greater than the speed at which the convolution block extracts the pitch feature vector, the convolution core of the shortcut convolution layer should be smaller than the convolution core of the 2D convolution layer of the convolution block, for example, the convolution core of the shortcut convolution layer may be 1×1.

[0047] After the shortcut convolution layer extracts and obtains the shortcut pitch features, stitching can be performed on the shortcut pitch features and the convergent pitch features output by the batch normalization layer, so as to combine the pitch features extracted based on different dimensions to obtain pitch stitching features. Then, the nonlinear relationship between the pitch stitching features is added by the relu activation function, and the voice stitching features are used as the input of the next convolution block, so as to obtain the pitch feature vector output by the next convolution block.

[0048] In some embodiments, the intermediate layer can include multiple ICB network structures, where the ICB network structures are similar to the REB network structures, and the only difference is that the ICB network structures do not have an average pooling layer. The intermediate layer can extract deeper pitch features in the pitch feature code through the ICB network structures, and output them in the format of the pitch feature code.

[0049] In some embodiments, the filter layer includes a multi-layer RCB convolutional network, and the RCB convolutional network in the filter layer may have the same dimensional arrangement as the REB network structure according to the input order, and the dimension of pitch feature extraction is sequentially increased to extract pitch features from shallow layers to deep layers.The above network structure in the embodiments of the present application obtains deep pitch features of the reference voice through a U-Net structure composed of a 2D convolutional layer and a deconvolutional layer, which facilitates improving the pitch similarity between the converted voice output by the subsequent voice conversion model and the voice of a real person.

[0050] In some embodiments, the decoder layer includes a multi-layer RDB network structure, as shown in FIG. 5, the RDB network structure includes a deconvolutional layer (transposed convolutional layer) and an RCB convolutional network, and one batch normalization layer and a Relu activation function can be connected between the deconvolutional layer and the RCB convolutional network. The roles of the batch normalization layer and the Relu activation function in the decoder layer can refer to their roles in the above encoder layer, and the embodiments of the present application will not be described.

[0051] In this embodiment, the decoder layer needs to perform decoding on the pitch feature code finally output by the intermediate layer and the filter layer, so it is necessary to first perform deconvolution calculation on the pitch feature code by a deconvolution layer to obtain a deconvolution feature vector, and then to improve the convergence speed of the decoder layer by a batch normalization layer, and to introduce nonlinear characteristics into the RCB convolution network in the decoder layer by a relu activation function.

[0052] After the batch normalization layer outputs the deconvolution feature vector, the RCB convolutional network performs decoding on the deconvolution feature vector hierarchically, thereby completing the extraction of the pitch feature of the reference voice.

[0053] In S200, feature stitching is performed on the audio feature code and the pitch feature to obtain an audio stitching feature.

[0054] In order to facilitate the combination of the pitch feature and the audio feature code, a stitching process can be performed on the audio feature code and the pitch feature to obtain the audio stitching feature of the reference audio, in which the pitch feature can be mapped to the audio feature code, thereby better combining the pitch feature with the audio feature code.

[0055] At S300, a linear spectrum corresponding to the reference speech is input to the post-encoder to obtain audio hidden variables.

[0056] In the learning and training process of the voice conversion model, a corresponding linear spectrum can be generated based on the reference voice, and then the linear spectrum can be input into a post-encoder, and the audio hidden variables corresponding to the reference voice can be output based on the post-encoder.

[0057] In addition, the audio hidden variables are generated by the post-encoder only in the learning training process, and are generated by the normalization flow in the pre-encoder of the voice conversion model in the application process, where the post-encoder can adopt the non-causal WaveNet residual module in WaveGlow and Glow-TTS, and the non-causal WaveNet residual module in WaveGlow and Glow-TTS is only applied in the learning training process and does not participate in the application process of the voice conversion model.

[0058] In S400, the timing alignment module aligns the timing sequence of the voice stitching feature and the audio latent variable to obtain a converted voice code.

[0059] Before performing voice conversion, it is necessary to ensure that the voice stitching feature and the audio latent variable are timing compatible, so as to alleviate the problem of video and audio not being synchronized between the converted voice and the reference voice. Therefore, the voice stitching feature and the audio latent variable can be input into a timing alignment module, which aligns the timing sequence of the voice stitching feature and the audio latent variable according to the timing alignment module, and outputs the converted voice code.

[0060] In some embodiments, a reference audio is further input to the projection layer of the pre-encoder, so that the audio stitching features of the reference audio are projected by the projection layer to the timing alignment module, so as to better combine the audio stitching features with the audio hidden variables, and improve the accuracy of the audio stitching features being fused into the converted audio code.

[0061] The timing alignment module may employ a Monotonic Alignment Search (MAS) alignment estimation algorithm, which is an algorithm used in audio signal processing to compare an audio sequence with a template, thereby performing an alignment operation.

[0062] Therefore, in some embodiments, in the process of inputting the voice stitching features and audio hidden variables into the timing alignment module, it is necessary to perform timing alignment for the voice stitching features and audio hidden variables based on a certain template voice sequence. In this embodiment, the template voice sequence of the timing alignment module can be obtained, and the template voice sequence is a voice sequence that speaks a specified text content with a preset speech rate, tone and pitch within a preset time. The template voice sequence plays a role of reference in the process of aligning the voice stitching features and audio hidden variables. The timing alignment module aligns the voice stitching features and audio hidden variables with the template voice sequence based on the MAS alignment estimation algorithm, and performs encoding on the aligned voice stitching features and audio hidden variables to obtain a converted voice code.

[0063] In some embodiments, before the audio stitching features and audio hidden variables are input into the timing alignment module, the audio stitching features and audio hidden variables are first input into a normalization flow, which improves the complexity of the prior distribution of the audio stitching features and audio hidden variables, thereby improving the complexity of the pitch stitching features, enhancing the efficiency of the voice conversion model to learn pitch features, and improving the pitch similarity between the converted voice output by the voice conversion model that has completed subsequent training and the voice of a real person.

[0064] In S500, the converted voice code is decoded by the decoder to obtain a converted voice.

[0065] In this embodiment, in order to better combine the voice stitching features of the reference voice, the voice stitching features of the code state and the audio hidden variables need to perform combination processing and timing alignment processing. However, since the voice conversion model cannot output the converted voice of the code state, the converted voice code needs to be input into the decoder of the voice conversion model, so that the converted voice code is decoded by the decoder to obtain the converted voice.

[0066] In this embodiment, the decoder may use the vocoder HiFi-GAN V1 generator or other vocoder generators, and the embodiment of the present application does not unduly restrict the type of vocoder used in the decoder.

[0067] In S600, a training loss of the converted voice is calculated, and if the training loss is less than or equal to a training loss threshold, a voice conversion model is output based on the current parameters of the trained model; if the training loss is greater than the training loss threshold, iterative training is performed on the voice conversion model.

[0068] After obtaining the converted voice, it indicates that the voice conversion model has completed one learning training process, and at this time, the training loss of the current converted voice can be calculated, and the learning training progress of the voice conversion model can be judged accordingly. As shown in Figure 6, if the training loss is less than the training loss threshold, it indicates that the voice conversion model has been trained to convergence, and at this time, the voice conversion model can be output according to the current parameters of the trained model.

[0069] In addition, the voice conversion model will reach a convergence state after completing learning training, so that the voice conversion model can enter the application stage. In order to easily distinguish between voice conversion models with different training levels, in the embodiments of the present application, the voice conversion model that has not been trained to convergence is defined as the trained model.

[0070] If the training loss is greater than the training loss threshold, it indicates that the voice conversion model has not been trained to converge, and at this time, it is necessary to continuously train the trained model with the reference voice, so that the trained model is repeatedly trained, and the training loss is calculated until the training loss is less than or equal to the training loss threshold after each training iteration, and the voice conversion model is trained to converge. In this way, in the application process, the voice conversion model can output a converted voice that has a high pitch similarity to a real human voice and has a high voice accuracy based on the target voice.

[0071] In some embodiments, the voice conversion model may further include a style encoder and a rhythm encoder, in which after performing feature stitching on the audio feature code and the pitch feature, the reference voice or a mel spectrum corresponding to the reference voice is further input into the style encoder to extract style features of the reference voice by the style encoder, and the obtained style features are mapped to the voice stitching features, thereby updating the voice stitching features, and the voice stitching features simultaneously hold the voice stitching features and style features of the reference voice.

[0072] In some embodiments, the style encoder can select the MelStyleEncoder module, which includes three sub-modules: Spectral processing, Temporal processing, and Multi-head attention.

[0073] Specifically, the structure and operation of the style encoder are as follows.

[0074] The spectral processing layer consists of one fully connected layer and is used to obtain the mel spectrum of the input training speech and convert it into a feature sequence.

[0075] The timing processing layer includes one gated convolution layer and one residual layer, and is used to obtain timing information in the feature sequence.

[0076] The attention layer is used to extract style features corresponding to the feature sequence corresponding to the timing information within a first preset time based on the timing information, and repeat this operation, where the first preset time is a frame-level short time, and the above operation is to respectively extract style features corresponding to the feature sequence within a plurality of short times. Based on this, at a second preset time, a plurality of style features corresponding to the plurality of first preset times are averaged to obtain a style vector, where the second preset time is usually a long time, and the second preset time includes the first preset time.

[0077] After inputting the Mel spectrum corresponding to the reference speech into the style encoder, the Spectral Processing sub-module in the style encoder can convert the input Mel spectrum into a sequence of frame-level hidden states using a fully connected layer. The Temporal Processing sub-module can extract timing information in the reference speech using Gated CNN and residual connections. The Multi-head attention sub-module can encode global information using a multi-head self-attention mechanism and residual connections, where the multi-head self-attention is used at the frame level to better extract style features from short reference speech and output style vectors (Style Embeddings) that are averaged over time.

[0078] In some embodiments, the style encoder may further include a style adaptive unit, which includes one normalization layer and one fully connected layer, and is used to predict corresponding feature biases and feature gains based on the style vector and output them as style information of the voice, and the output style information of the voice is used for subsequent voice conversion. In the conventional voice conversion process, subsequent operations are performed based only on style features obtained based on the audio, and a large number of training samples are required to achieve high results, so more accurate style extraction can be achieved. With the above improvement, the style information changes adaptively based on the change of the style vector, making the reproduction of the style more accurate and reducing the demand for training voice.

[0079] In some embodiments, the pre-encoder may further include a text encoder, which may cooperate with the rhythm encoder to extract rhythm features of the reference audio. In this embodiment, the reference text may be input to the text encoder, i.e., the reference text is a text corresponding to the audio content in the reference audio, and the text encoder may extract text features based on the reference text. In addition, the text features may be input to the rhythm encoder together with the reference audio, and the rhythm encoder may output rhythm features based on the text features and the reference audio.

[0080] In this embodiment, the rhythm encoder can use a ProsodyEncoder module, which can extract rhythm features from the reference audio through a word-level vector quantization bottleneck.

[0081] In some embodiments, as shown in Figure 7, the rhythm encoder includes a rhythm convolution layer and a pooling layer, where the rhythm convolution layer includes a relu activation function and a normalization layer, and the relu activation function can eliminate the linearization of the rhythm encoder, thereby providing the rhythm convolution layer with nonlinear expression ability, fitting deeper rhythm features, and improving the accuracy of extracting rhythm features. The normalization layer is used to perform normalization processing on each reference voice sample, which improves the convergence speed of the rhythm encoder, reduces the phenomenon of overfitting, and improves the efficiency of extracting rhythm features.

[0082] The structure of the rhythm encoder may be one rhythm convolution layer, a pooling layer, and another rhythm convolution layer. For convenience of description, the embodiments of the present application define the rhythm convolution layer before the pooling layer as the first rhythm convolution layer, and the rhythm convolution layer after the pooling layer as the second rhythm convolution layer.

[0083] In the process of inputting the text features and the reference speech into the rhythm encoder, the text features are directly input into the rhythm convolution layer, so that the rhythm convolution layer compresses the text features and obtains word-level hidden features from the text features. The reference speech is first preprocessed into a Mel spectrum, and the Mel spectrum is input into the first rhythm convolution layer, the pooling layer and the second rhythm convolution layer in sequence according to the structure of the rhythm encoder. After the Mel spectrum of the reference speech is input into the first rhythm convolution layer, the first rhythm convolution layer can perform word-level rhythm quantization on the reference speech based on the compressed word-level hidden features to obtain rhythm features.

[0084] After the first rhythm convolution layer outputs the rhythm features, the pooling layer performs feature dimensionality reduction on the word-level hidden features and rhythm attribute features to obtain rhythm codes, which can reduce the computational complexity of the speech conversion model, reduce the overfitting problem, and improve the feature extraction efficiency. After the pooling layer outputs the rhythm codes, the rhythm codes can be input to the second rhythm convolution layer to extract deep rhythm features, and finally the deep rhythm features can be compressed by a vector quantization layer to output the final rhythm features, improve the feature extraction accuracy, and make the rhythm codes obtained in the training process of the speech conversion model more consistent with the real human speech rhythm.

[0085] In some embodiments, the audio stitching feature, style feature and rhythm feature can be input into the pre-encoder normalization flow simultaneously, thereby improving the complexity of the audio stitching feature, style feature and rhythm feature. And based on the timing alignment module, the audio stitching feature, style feature and rhythm feature after the complexity improvement and the audio hidden variable are subjected to timing alignment, and the obtained audio code is simultaneously combined with the audio stitching feature, style feature and rhythm feature to reduce the audio difference between the converted voice and the real human voice.

[0086] In some embodiments, the audio encoder can be built based on a pre-trained Hubert model, and the pre-training for the audio encoder can enable the audio encoder to better extract the audio features of the reference voice. The pre-training process is described below.

[0087] First, a clustering model constructed based on a k-means network is preset, and the clustering model can include a feature extraction layer, a clustering processing layer, and a category labeling layer. Here, the feature extraction layer is used to perform feature extraction on the training data in the pre-training process, and a self-supervised model, for example, the above-mentioned Hubert model, is used. The feature extraction layer is also used to configure the feature extraction part of the audio encoder. The clustering processing layer is constructed based on the K-mens model, and is used to perform clustering processing on the extracted audio features, that is, to cluster audio features of the same category. The category labeling layer is used to assign a category code corresponding to the audio features of a certain category after clustering.

[0088] After completing the construction of the clustering model, the clustering model can be pre-trained by using generic training data, and the generic training data can be based on LibriSpeech-960 and AISHELL-3 data. For example, 200 speakers' voice sample data is obtained, and the number of clusters is 200. In the clustering model, the feature extraction layer performs audio feature processing on the training data, and the clustering processing layer performs clustering on the corresponding audio features, so that the clustering model can perform clustering processing on the voice samples of different speech categories. Note that the clustering processing means that voices with similar voice styles are automatically clustered by unsupervised training and similarity calculation, and voices of the same category may not be the same speaker, and it is only necessary that the similarity of the voice style reaches a preset similarity threshold.

[0089] The category label layer can perform category encoding on the clustered categories for the subsequent training of the audio encoder. For example, different categories can be obtained after clustering the voice sample data by the clustering model, and the different categories can be given category codes of ID1.1, ID1.2, ... ID1.9, respectively. The category codes allow each audio feature category to have a unique label for distinction after clustering by the clustering model, which makes it easy to map and encode the categories in the subsequent learning and training process of the voice conversion model.

[0090] In addition, in the above clustering model, other than the feature extraction layer, the remaining part does not participate in the construction of the audio encoder, and only provides the audio feature category code to the audio encoder during the training stage of the audio encoder. After the voice conversion model completes training, the clustering model also does not participate in the inference operation in the application process of the voice conversion model.

[0091] After the audio encoder completes pre-training, the audio encoder can be constructed based on the model and each layer structure constructed after pre-training, and in some embodiments, the audio encoder includes the following parts:

[0092] The feature code unit is constructed by the Hubert model that has completed the clustering pre-training in the clustering model, and is used to extract and encode the audio features of the reference speech.

[0093] The category mapping unit is composed of one mapping layer and is used for mapping audio feature category codes, i.e., for mapping corresponding category codes to audio features extracted by the feature code unit.

[0094] The category code unit is composed of one embedded layer and is used to assign category codes defined in the clustering model to audio features extracted by the feature code unit during the training process of the audio encoder.

[0095] In the training process of the audio encoder, the feature coding unit and the category mapping unit are first initialized, that is, some parameters of the Hubert model and the mapping layer are randomly initialized. After the initialization is completed, the audio encoder can be trained by using generic training data.

[0096] In the actual training process of the audio encoder, in addition to performing model training and parameter updating normally, the audio encoder can also be trained based on the predicted category codes of the category codes corresponding to the general-purpose training data and the real category codes obtained by the clustering model.

[0097] A category code is mapped to the audio feature extracted by the feature extraction unit by a category mapping unit and a category code unit, and the category code is a predicted category code for the generic training data. Then, a minimization process is performed on the average cross entropy between the predicted category code of the Hubert model for the generic training sample and the real category code given in the training process of the clustering model, and the loss function of the audio encoder is updated according to the result, and the relevant parameters of the audio encoder are updated accordingly, so as to complete the training of the audio encoder.

[0098] In some embodiments, various training losses may be included in the process of training the voice conversion model, and therefore, in the embodiments of the present application, the training losses may include spectrum loss, variance loss, decoder loss, random duration prediction loss, and feature matching loss when training the generator, and in this embodiment, the training loss may be expressed by the following formula:

number

[0099] In some embodiments, the spectral loss is the training loss between the training voice and the training synthetic voice. To calculate the spectral loss, the spectral accuracy of the training voice is obtained, and the spectral accuracy of the training synthetic voice is obtained, and the spectral loss can be calculated based on the spectral accuracy of the training voice and the spectral accuracy of the training synthetic voice by the following formula:

number

[0100] In some embodiments, L kl is the KL variance loss, which is a loss between the posterior distribution estimate of the final hidden variables obtained by combining the variables obtained by processing the linear spectrum of the audio by the posterior encoder with the style vector output by the style encoder and the rhythm code output by the rhythm encoder, and the prior distribution estimate of the hidden variables between the given condition text and its information.

[0101] In this embodiment, a posterior distribution result is calculated based on the audio hidden variables, alignment information for synthesizing a voice code is obtained, and then a prior distribution result is calculated based on the alignment information and the audio hidden variables, and finally the variance loss can be calculated based on the posterior distribution result and the prior distribution result by the following formula:

number

[0102] In some embodiments, the decoder further includes a classifier, which can classify the synthetic voice obtained after decoding in the process of applying the voice conversion model, and if the classifier cannot distinguish between the synthetic voice and the voice of a real person, it indicates that the accuracy of the synthetic voice has reached the accuracy of the voice of a real person, and thus outputs the synthetic voice.

[0103] In the training process of the speech conversion model, the training synthetic speech is input to a classifier to obtain the discrimination features output by the classifier, and the feature matching loss can be calculated when training the generator, where the feature matching loss can be regarded as a reconstruction loss and is used to constrain the output of the intermediate layer of the classifier, and the classifier is used to perform adversarial training with the decoder in the training speech conversion model.

[0104] In some embodiments, the feature matching loss when training the generator can be calculated by the following formula:

number

[0105] In some embodiments, L adv ×L fm (G) is the loss of the decoder module, where L advis the least squares loss function for adversarial training, and the loss of the classifier can be calculated by the following formula:

number

[0106] The generator losses can be calculated using the following formula:

number

[0107] Therefore, the least squares loss function of the adversarial training can be calculated based on the discriminator loss and the generator loss, and the decoder loss can be calculated based on the least squares loss function of the adversarial training and the feature matching loss when training the generator.

[0108] In some embodiments, L dur is the random time length prediction loss. Through the timing alignment module, the text code of the training text is obtained based on the MAS algorithm, and then the predicted mean variance and hidden variable Z are processed through the normalization flow to obtain the optimal alignment matrix of normal distribution, and the random time length prediction loss can be calculated.

[0109] As can be seen from the above technical solutions, the present application provides a pitch-based voice conversion model training method and a voice conversion system, the method being used to train a voice conversion model, the voice conversion model including a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module. The method includes inputting a reference voice into a pre-encoder and a pitch extraction module, so that the pre-encoder outputs an audio feature code, and the pitch extraction module extracts a pitch feature. Then, the linear spectrum corresponding to the reference voice is input into the post-encoder to obtain an audio hidden variable. The audio feature code and the pitch feature are stitched to obtain an audio stitching feature and an audio hidden variable, which are input into a timing alignment module to obtain a converted voice code, and the converted voice code is decoded by a decoder to obtain a converted voice. The training loss of the converted voice is then calculated to determine the convergence degree of the voice conversion model. The present application uses a pitch extraction module to extract pitch features of the reference voice, stitches them with audio feature codes, and performs alignment so that the pitch features of the converted voice are more consistent with the real voice of a person, and improves the pitch similarity of the converted voice when voice samples are insufficient.

[0110] References throughout this specification to "embodiments," "some embodiments," "an embodiment," or "an embodiment" or the like mean that a particular feature, element, or characteristic described with reference to this embodiment is included in at least one embodiment, and thus, the collocations "in embodiments," "in some embodiments," "in at least another embodiment," or "in an embodiment" or the like referred to throughout this specification do not necessarily refer to the same embodiment. Furthermore, particular features, elements, or characteristics may be combined in any suitable manner in one or more embodiments. Thus, without limitation, a particular feature, element, or characteristic illustrated or described with reference to one embodiment may be combined in whole or in part with a feature, element, or characteristic of one or more other embodiments. Such modifications and variations are within the scope of this application.

[0111] Similar parts between the embodiments provided in this application may be mutually referenced, and the specific embodiments provided above are only some examples of the overall concept of this application, and do not limit the scope of protection of this application. Any other embodiments extended based on the solutions of this application without creative effort by those skilled in the art are also within the scope of protection of this application.

[0112] What is described above is only a preferred embodiment of the present application, and it should be noted that those skilled in the art may make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be considered as within the protection scope of the present application.

Claims

1. A method for training a voice conversion model, the voice conversion model including a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module, the pitch extraction module including an encoder layer, a filter layer, an intermediate layer and a decoder layer, the encoder layer, the intermediate layer and the decoder layer forming a first encoding branch of the pitch extraction module, and the encoder layer, the filter layer and the decoder layer forming a second encoding branch of the pitch extraction module; A method for training a pitch-based voice conversion model, comprising: the encoder layer includes an average pooling layer and a convolutional network; the convolutional network includes a convolutional block and a shortcut convolutional layer; the convolutional block includes a 2D convolutional layer, a batch normalization layer and a relu function; and the shortcut convolutional layer is disposed between an input end of the convolutional network and an output end of the convolutional network, inputting a reference voice into the pre-encoder and the pitch extraction module, extracting audio feature codes by the pre-encoder, and extracting a pitch feature vector of the reference voice by the convolutional network; performing downsampling on the pitch feature vector by the average pooling layer to obtain a pitch feature code; performing decoding on the pitch feature code by the decoder layer to obtain pitch features; performing feature stitching on the audio feature code and the pitch feature to obtain an audio stitching feature; inputting a linear spectrum corresponding to the reference speech into the post-encoder to obtain audio hidden variables; Aligning the timing sequence of the audio stitching feature and the audio latent variable by the timing alignment module to obtain a converted audio code; decoding the converted speech code by the decoder to obtain a converted speech; calculating a training loss of the converted speech, and if the training loss is less than or equal to a training loss threshold, outputting a speech conversion model based on current parameters of a trained model, and if the training loss is greater than the training loss threshold, performing iterative training on the speech conversion model, where the trained model is a speech conversion model that has not been trained to convergence; The pitch feature vector is Extracting deep audio vectors using the 2D convolutional layer; performing a fast convergence process on the deep audio vectors by the batch normalization layer to extract convergent pitch features from the deep audio vectors; adding a nonlinear relationship to the convergent pitch feature by the relu function to obtain the pitch feature vector; Or, before the step of adding a non-linear relationship to the convergent pitch feature by the relu function, extracting shortcut pitch features by the shortcut convolutional layer, the shortcut convolutional layer including only one 2D convolutional layer, a speed at which the shortcut convolutional layer extracts the shortcut pitch features is greater than a speed at which the convolutional block extracts the pitch feature vector, and the shortcut pitch features are pitch features extracted by the shortcut convolutional layer; stitching the shortcut pitch feature and the convergent pitch feature to obtain a pitch stitching feature; adding a nonlinear relationship between the pitch stitching features by the relu function to obtain the pitch feature vector.

2. The decoder layer includes a deconvolution layer and the convolution network, and the step of performing decoding on the pitch feature code by the decoder layer includes: performing a deconvolution calculation on the pitch feature code by the deconvolution layer to obtain a deconvolution feature vector; and performing decoding on the deconvolved feature vector by the convolutional network to obtain the pitch features.

3. The step of aligning the timing sequences of the audio stitching features and the audio latent variables by the timing alignment module includes: obtaining a template speech sequence for a timing alignment module; aligning timing sequences of the audio stitching features and the audio latent variables based on the template audio sequence; and performing encoding on the aligned phonetic stitching features and the audio hidden variables to obtain the converted phonetic code.

4. The voice conversion model further includes a style encoder, and after the step of performing feature stitching on the audio feature codes and the pitch features, extracting style features of the reference audio by the style encoder; 2. The method of claim 1, further comprising: mapping the style features to the phonetic stitching features to update the phonetic stitching features.

5. The training loss includes a spectral loss, and the step of calculating the training loss of the converted speech includes: obtaining a spectral accuracy of the reference speech and obtaining a spectral accuracy of the converted speech; Calculating the spectral loss based on the spectral accuracy of the reference speech and the spectral accuracy of the converted speech according to the following formula: [0010] Here, L ｒｅｃｏｎ is the spectral loss, and x ｍｅｌ is the spectral accuracy of the reference voice, and x ＾ ｍｅｌ 2. The method of claim 1, further comprising the steps of: determining whether the pitch-based speech conversion model is a pitch-based speech conversion model; and determining whether the pitch-based speech conversion model is a pitch-based speech conversion model.

6. A voice conversion model, the voice conversion model being obtained by training based on the pitch based voice conversion model training method according to any one of claims 1 to 5, the voice conversion model comprising a pre-encoder, a post-encoder, a timing alignment module, a decoder and a pitch extraction module, the pitch extraction module comprising an encoder layer, a filter layer, an intermediate layer and a decoder layer, the encoder layer, the intermediate layer and the decoder layer forming a first encoding branch of the pitch extraction module, the encoder layer, the filter layer and the decoder layer forming a second encoding branch of the pitch extraction module, A voice conversion system, comprising: the encoder layer includes an average pooling layer and a convolutional network; the convolutional network includes a convolutional block and a shortcut convolutional layer; the convolutional block includes a 2D convolutional layer, a batch normalization layer and a relu function; and the shortcut convolutional layer is disposed between an input end of the convolutional network and an output end of the convolutional network; The pre-encoder is configured to extract audio feature codes of a reference audio; the pitch extraction module is configured to extract a pitch feature vector of the reference voice by the convolutional network, perform downsampling on the pitch feature vector by the average pooling layer to obtain a pitch feature code, and perform decoding on the pitch feature code by the decoder layer to obtain a pitch feature; The post-encoder is configured to generate audio hidden variables based on a linear spectrum corresponding to the reference audio; The timing alignment module is configured to align a timing sequence of the audio stitching feature and the audio latent variable to obtain a converted audio code, and the audio stitching feature is obtained by combining the audio feature code and the pitch feature; the decoder is configured to perform decoding on the converted audio code to obtain a converted audio; The pitch feature vector is Extracting deep audio vectors using the 2D convolutional layer; performing a fast convergence process on the deep audio vectors by the batch normalization layer to extract convergent pitch features from the deep audio vectors; adding a nonlinear relationship to the convergent pitch feature by the relu function to obtain the pitch feature vector; Or, before the step of adding a non-linear relationship to the convergent pitch feature by the relu function, extracting shortcut pitch features by the shortcut convolutional layer, the shortcut convolutional layer including only one 2D convolutional layer, a speed at which the shortcut convolutional layer extracts the shortcut pitch features is greater than a speed at which the convolutional block extracts the pitch feature vector, and the shortcut pitch features are pitch features extracted by the shortcut convolutional layer; stitching the shortcut pitch feature and the convergent pitch feature to obtain a pitch stitching feature; adding a nonlinear relationship between the pitch stitching features by the relu function to obtain the pitch feature vector.

Images