Speech conversion model training method and device, equipment and medium

Abstract

The invention discloses a voice conversion model training method, device and system and a medium, and the method comprises the steps: an encoder processes an input Mel spectrum, outputs speaker embedding, content embedding and implicit coding, and inputs the speaker embedding, the content embedding and the implicit coding to a decoder; after the decoder receives speaker embedding, content embedding and implicit coding, a reconstructed Mel spectrum is generated through feature fusion and up-sampling operation; and calculating a total loss value through the target Mel spectrum, speaker embedding, content embedding and the reconstructed Mel spectrum, and updating model parameters according to the total loss value. Speaker embedding, content embedding and implicit coding are obtained through the encoder, and the decoder reconstructs the Mel spectrum. The model is trained according to the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum calculation loss value, so that the damage and loss of audio information are effectively avoided, the ability of capturing and simulating the speech features of the speaker of the model is improved, and the naturalness of speech conversion is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech data processing technology, which is applied to dialogue scenarios such as finance or medicine, and in particular to a training method, device, equipment and storage medium for a speech conversion model. Background Art

[0002] Speech conversion technology is a technology that can convert the speech content of a source speaker into speech that retains the speech content while having the timbre and intonation characteristics of the target speaker. However, using vector quantization to decouple speech features may lose some important audio features, resulting in a decrease in the quality of the converted speech, which may sound unnatural or somewhat mechanical. And when changing the application scenario or speaker, it is usually necessary to collect data again and train the model, which is not only time-consuming but also costly. Traditional models may perform well on specific datasets, but their performance deteriorates on new or different datasets and lack sufficient generalization capabilities. And traditional methods may have difficulty accurately capturing and replicating all the speech features of the target speaker, including subtle differences such as emotions and speaking speed. The performance of traditional models depends largely on the quality and diversity of training data. If the training data is insufficient or biased, the model may not be able to learn and convert well. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to propose a training method, device, equipment and storage medium for a speech conversion model to solve the problem that audio feature damage and loss occur when the conventional technology uses traditional models for speech feature decoupling, and to improve the naturalness and adaptability of the speech conversion model.

[0004] In order to solve the above technical problems, the embodiment of the present invention provides a training method for a speech conversion model, which adopts the following technical solution:

[0005] A method for training a speech conversion model comprises the following steps:

[0006] The encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder;

[0007] After receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations;

[0008] The total loss value is calculated through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, and the model parameters are updated according to the total loss value to complete the speech conversion model training.

[0009] In one embodiment, the encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder, including:

[0010] Acquire training speech data, and pre-process the training speech data to obtain a training data set;

[0011] The training speech data set is converted into mel-spectrogram.

[0012] In one embodiment, the encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder, including:

[0013] The encoder extracts features from the mel spectrum to obtain latent codes;

[0014] Performing a vector quantization operation on the latent code to generate a content embedding;

[0015] Calculating the difference between the latent code and the content embedding to obtain speaker embedding;

[0016] The speaker embedding, content embedding and latent coding are input to the decoder.

[0017] In one embodiment, after receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations, including:

[0018] The decoder performs feature fusion on the speaker embedding and the content embedding to obtain a feature embedding;

[0019] The feature embedding is added to the latent code, and up-sampling is performed to obtain a reconstructed Mel spectrum.

[0020] In one embodiment, the calculating of the total loss value through the target mel spectrum, the speaker embedding, the content embedding and the reconstructed mel spectrum includes:

[0021] Calculating a reconstruction loss according to the Mel spectrum and the reconstructed Mel spectrum;

[0022] Input the speaker embedding and content embedding into the corresponding classifier to calculate the classification loss;

[0023] Calculate the quantization loss when performing vector quantization operations on the hidden code;

[0024] The total loss value is calculated by weighted summation of reconstruction loss, classification loss, and quantization loss.

[0025] In one embodiment, updating the model parameters according to the total loss value to complete the speech conversion model training includes:

[0026] Use the back-propagation algorithm to calculate the gradient of the model parameters based on the total loss value;

[0027] The model parameters are updated according to the gradient through the optimization algorithm to complete the speech conversion model training.

[0028] In one embodiment, after the steps of calculating the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, updating the model parameters according to the total loss value and completing the speech conversion model training, the method further includes:

[0029] The encoder obtains the target speaker embedding based on the target speaker’s Mel-spectrogram.

[0030] The encoder obtains the source speaker content embedding and the source speaker latent code according to the source speaker's mel spectrum;

[0031] The source speaker content embedding, the source speaker hidden code and the target speaker embedding are input into the corresponding level decoder for information fusion and upsampling operations, and the characteristic Mel spectrum of the target speaker is output to complete the speech conversion.

[0032] In order to solve the above technical problems, the embodiment of the present invention also provides a training device for a speech conversion model, which adopts the following technical solution:

[0033] A training device for a speech conversion model, comprising:

[0034] An encoder module, configured to process the input mel spectrum by an encoder, output speaker embedding, content embedding and latent code, and input the speaker embedding, content embedding and latent code to a decoder;

[0035] A decoder module, configured to generate a reconstructed mel spectrum through feature fusion and upsampling operations after the decoder receives the speaker embedding, content embedding and latent coding;

[0036] The model training module is used to calculate the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, update the model parameters according to the total loss value, and complete the speech conversion model training.

[0037] In order to solve the above technical problems, an embodiment of the present invention further provides a computer device, which adopts the following technical solution:

[0038] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech conversion model training method when executing the computer-readable instructions.

[0039] In order to solve the above technical problems, an embodiment of the present invention further provides a computer-readable storage medium, which adopts the following technical solution:

[0040] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the training method for a speech conversion model as described above.

[0041] Compared with the prior art, the embodiments of the present invention mainly have the following beneficial effects:

[0042] The training method of the speech conversion model described in the embodiment of the present invention processes the input mel spectrum by the encoder, outputs speaker embedding, content embedding and hidden code, and inputs the speaker embedding, content embedding and hidden code to the decoder; after the decoder receives the speaker embedding, content embedding and hidden code, it generates a reconstructed mel spectrum through feature fusion and upsampling operations; calculates the total loss value through the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum, and updates the model parameters according to the total loss value. The present invention obtains the speaker embedding, content embedding and hidden code through the encoder, and the decoder reconstructs the mel spectrum. The loss value is calculated according to the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum to train the model, effectively avoiding damage and loss of audio information, improving the model's ability to capture and imitate the speaker's voice features, and the naturalness of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 is an exemplary system architecture diagram in which the present invention can be applied;

[0045] Figure 2 is a flow chart of an embodiment of a method for training a speech conversion model according to the present invention;

[0046] Figure 3 is a flow chart of an embodiment of a method for converting a speech conversion model according to the present invention;

[0047] Figure 4 is a flow chart of an embodiment of a method for implementing a conversion according to a speech conversion model of the present invention;

[0048] Figure 5 is a structural schematic diagram of an embodiment of a training device for a speech conversion model according to the present invention;

[0049] Figure 6 FIG. 1 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0054] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0055] Terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Dynamic Image Experts Compression Standard Audio Layer 3),

[0056] MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops and desktop computers, etc.

[0057] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0058] It should be noted that the training method of the speech conversion model provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the training device of the speech conversion model is generally arranged in the server / terminal device.

[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0060] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for training a speech conversion model according to the present invention. The method for training a speech conversion model comprises the following steps:

[0061] S1. The encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder;

[0062] S2, after receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations;

[0063] S3. Calculate the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, update the model parameters according to the total loss value, and complete the speech conversion model training. In this embodiment, the vector quantization and normalization methods are used to decouple the feature vector, and the speaker information (speaker embedding) and content information (content embedding) contained in the speech data are successfully separated. Specifically, speaker information refers to information related to individual speaker characteristics in speech, such as: unique timbre, intonation, and emotional characteristics; while content information corresponds to the energy size of speech. In order to reduce the loss of quantization to the quality of speech audio data, the multi-layer structure of Unet is used to improve the quality of audio synthesis. At the same time, a classifier and a gradient reversal layer are introduced to guide the model body to generate content embedding independent of the speaker and a more expressive speaker embedding through the gradient transmitted in reverse by the classifier.

[0064] Specifically, the encoder module will first perform feature extraction and dimensionality reduction operations on the input Mel spectrum to generate a hidden code. The hidden code is then used to generate a content embedding through instance normalization and vector quantization, and the hidden code is subtracted from the content embedding to obtain the speaker embedding. The speaker embedding and content embedding are input to the decoder module of the corresponding level, and the hidden code is input to the next encoder module. After hierarchical processing, the decoder outputs the reconstructed Mel spectrum. The speaker embedding is input into the speaker classifier to calculate the loss, and the content embedding is input into the speaker classifier with a gradient reversal layer to calculate the loss. The total loss value is calculated using the target Mel spectrum, speaker embedding, content embedding, and reconstructed Mel spectrum, and the model parameters are updated according to the total loss value to complete the speech conversion model training.

[0065] In one embodiment, the encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder, including:

[0066] Acquire training speech data, and pre-process the training speech data to obtain a training data set;

[0067] The training speech data set is converted into mel-spectrogram.

[0068] In this embodiment, speech data for training is collected, wherein the speech data includes publicly available data sets, or speech data recorded according to training tasks. The speech signal is recognized and segmented into effective speech parts, and silent or non-speech parts are removed. The speech data generates an analog audio signal. The analog audio signal is digitized and converted into a digital signal through an analog-to-digital converter. The digitized speech signal (specifically the sound made by humans, which is a type of analog audio signal. The speech signal can be converted into digital audio waveform data or processed as an analog signal) is preprocessed. The processing includes denoising, segmentation, and normalization, and the processed data is formed into a data set to improve the quality of the speech data.

[0069] Specifically, denoising is done by reducing background noise and other interference by using filters or denoising algorithms. Segmentation is done by breaking up long recordings into shorter segments corresponding to individual words, phrases, or sentences. Normalization is done by adjusting the amplitude of the signal so that it has a uniform dynamic range, usually by scaling the signal to a fixed maximum value or using normalization techniques.

[0070] Furthermore, a fast Fourier transform (FFT) is performed on each frame of the speech signal to convert the speech signal from the time domain to the frequency domain. The result of the FFT is converted to the Mel scale. Mel-spectrogram is a spectrum representation method commonly used in speech signal processing and music information retrieval. It maps the frequency range of the signal to the Mel scale based on the perceptual characteristics of the human auditory system. Among them, the Mel scale is a logarithmic scale that is more in line with human auditory perception. The output energy of each Mel filter is calculated, and the logarithm of the output energy is taken to obtain the Mel spectrum.

[0071] Specifically, the Mel spectrum is divided into short frames, and a window function is applied to each frame of speech signal. First, the frame length (usually between 25-40 milliseconds) and the frame shift (i.e., the time interval between two consecutive frames) are determined. The frame shift is usually smaller than the frame length to ensure that there is enough overlap between frames, which helps to capture the dynamic characteristics of the signal. The Mel spectrum is divided according to the selected frame length to obtain a series of short frames. Each short frame represents the spectral characteristics of the signal within a specific time window. A window function is applied to each short frame, wherein the window function is used to modify the amplitude of the signal so as to more accurately analyze the signal in the frequency domain. For example, the rectangular window, the simplest window function, has the same weight for all samples; the Hamming window, adds a weight factor on the basis of the rectangular window. This embodiment recommends using a Hamming window to process the speech signal to reduce the effect of discontinuity at the signal frame boundary, thereby reducing the spectrum leakage effect.

[0072] Perform a fast Fourier transform (FFT) on each short frame to which the window function is applied to obtain a spectral representation of each short frame. Fast Fourier transform can provide local characteristics of the signal in the time-frequency plane. The result of the fast Fourier transform is mapped to the Mel scale to obtain a Mel spectrum. The Mel spectra of all short frames are arranged in chronological order to form a Mel spectrogram. The Mel spectrogram represents the Mel frequency in the vertical direction and the time in the horizontal direction. Mel spectrogram segmentation and the application of window functions are key steps in speech signal processing. They help to extract the time-frequency features of the signal and provide useful information for subsequent tasks such as speech recognition and sentiment analysis.

[0073] In one embodiment, the encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder, including:

[0074] The encoder extracts features from the mel spectrum to obtain latent codes;

[0075] Performing a vector quantization operation on the latent code to generate a content embedding;

[0076] Calculating the difference between the latent code and the content embedding to obtain speaker embedding;

[0077] The speaker embedding, content embedding and latent coding are input to the decoder.

[0078] In this embodiment, the encoder receives a mel spectrum as input, which is a frequency domain representation extracted from the original speech signal. Mel spectrum is a frequency distribution diagram that shows the characteristics of an audio signal. The construction of mel spectrum is based on human perception of frequency, so using mel spectrum as input data can enable the model to generate high-quality audio that satisfies human subjective evaluation. The encoder extracts features through a series of processing steps (such as convolutional layers, recurrent layers, or attention mechanisms) to obtain hidden codes. The hidden code is input into a codebook for vector quantization operation, and the hidden code is mapped to a discrete codebook through vector quantization operation to generate content embedding. The codebook contains a limited number of discrete codes, and the L2 norm of the hidden code and each discrete code is calculated, and the discrete code corresponding to the minimum L2 norm is selected as the quantization result output of the hidden code. Quantization can extract most of the speech energy information, that is, content embedding. Content embedding corresponds to the content information of the speech. The hidden code is subtracted from the content embedding, and the continuous vector obtained by the subtraction captures information that is not easy to quantize or related to the characteristics of individual speakers, that is, speaker embedding. The speaker embedding, content embedding, and hidden code are input to the decoder.

[0079] In one embodiment, after receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations, including:

[0080] The decoder performs feature fusion on the speaker embedding and the content embedding to obtain a feature embedding;

[0081] The feature embedding is added to the latent code, and up-sampling is performed to obtain a reconstructed Mel spectrum.

[0082] In this embodiment, the speaker embedding and content embedding of the corresponding level and the latent encoding decoder module output in the previous step. The decoder module receives the speaker embedding, content embedding and latent encoding. Speaker embedding contains speaker information, and content embedding contains content information, using embedding as a "carrier" to store the corresponding information. These two embeddings capture the speaker characteristics and content information of the speech respectively. Speaker embedding (Speaker Embedding) and content embedding (ContentEmbedding) are input into the decoder for feature fusion. Among them, the fusion operation is to add the content embedding to the speaker embedding and then implement it through the neural network. Feature fusion is usually implemented by simple addition or more complex operations (such as element-by-element addition, layer-by-layer fusion or fusion through neural network layers). Add the fused feature embedding to the latent encoding (Latent Encoding). The latent encoding contains a compressed representation of the original speech signal, which helps to maintain the integrity of the reconstructed speech. The result after addition is upsampled in the time domain and frequency domain. To gradually increase the dimension and time resolution of the data. Upsampling can be transposed convolution, upsampling layer or interpolation method, etc. After upsampling, the decoder outputs the reconstructed Mel spectrum, which is the spectral representation of the speech signal and can be used to generate an audible speech waveform. Finally, the reconstructed Mel spectrum is obtained by upsampling through the hierarchical decoder.

[0083] In one embodiment, the calculating of the total loss value through the target mel spectrum, the speaker embedding, the content embedding and the reconstructed mel spectrum includes:

[0084] Calculating a reconstruction loss according to the Mel spectrum and the reconstructed Mel spectrum;

[0085] Input the speaker embedding and content embedding into the corresponding classifier to calculate the classification loss;

[0086] Calculate the quantization loss when performing vector quantization operations on the hidden code;

[0087] The total loss value is calculated by weighted summation of reconstruction loss, classification loss, and quantization loss.

[0088] In this embodiment, when performing speech synthesis or speech conversion tasks, the loss function is usually composed of multiple parts, each of which is optimized for different aspects of the model. Reconstruction loss (Reconstruction Loss) is calculated based on the Mel spectrum and the reconstructed Mel spectrum. Use an appropriate loss function to measure the difference between the two. A common choice is the mean square error (MSE). The speaker embedding and content embedding are input into the corresponding classifier, and the output of the classifier is a probability distribution, and the classification loss (Classification Loss) is calculated. The cross entropy loss is used to measure the difference between the probability distribution of the model output and the true label. In vector quantization, the hidden code is quantized into a discrete vector representation, and the quantization loss measures the difference between the vector before and after quantization, and the quantization loss (Quantization Loss) when the hidden code is vector quantized is calculated. The quantization loss can be calculated by MSE or other metrics.

[0089] Determine all the loss items that need to be optimized in the model, assign a weight to each loss item, and the weight reflects the importance of each loss item in the total loss. The loss weight can be adjusted according to the importance of the task to improve the fairness and accuracy of the model. The total loss value Total Loss is calculated by weighted summation of reconstruction loss, classification loss, and quantization loss.

[0090] Total Loss=λ1Reconstruction Loss+λ2Classification Loss+

[0091] λ3Quantization Loss,

[0092] Where λ1, λ2, and λ3 are the weights of the corresponding losses. Use the total loss to backpropagate the model parameters and calculate the gradient. Use an optimization algorithm (such as SGD, Adam, etc.) to update the model parameters according to the gradient.

[0093] In this way, the model can simultaneously learn to reconstruct speech features, distinguish different speakers and content, and optimize the representation of the latent code. The design of the total loss function is crucial to the training effect of the model and needs to be carefully adjusted to balance the contributions of different parts. Weighted summation is a mathematical operation in which each value is multiplied by a specific weight and then these products are added to get the final sum. Weighted summation is usually expressed as

[0094]

[0095] where w i is the i-th value x iwhere n is the weight of the sum and n is the total number of values. A weight is a number that represents the importance or influence of each value in the sum. Weights can be positive or negative, but positive values ​​are usually used in sums. Weighted sums are used to calculate weighted averages, where each data point is given a different weight based on its importance.

[0096] In finance, weighted sums can be used to calculate the weighted average cost of capital. In machine learning, weighted sums are used in the calculation of loss functions, where different types of errors may be given different weights depending on their severity. In feature engineering, weighted sums can be used to combine the effects of different features to form a more powerful predictive model. When performing weighted sums, it is important to be aware of numerical stability, especially when the weights or values ​​are very large or very small. The sum of the weights is usually normalized to 1 to ensure that the result of the weighted sum can be interpreted as an average or proportion. In this way, it is possible to ensure that the model makes balanced progress on multiple objectives rather than just optimizing a single loss function. This is very important in practical applications, especially in complex tasks that require the model to perform well in multiple aspects.

[0097] In one embodiment, updating the model parameters according to the total loss value to complete the speech conversion model training includes:

[0098] Use the back-propagation algorithm to calculate the gradient of the model parameters based on the total loss value;

[0099] The model parameters are updated according to the gradient through the optimization algorithm to complete the speech conversion model training.

[0100] In this embodiment, first, forward propagation is performed, that is, starting from the input layer, layer by layer calculation until the output layer, to obtain the prediction result. Use a loss function (such as mean square error, cross entropy, etc.) to calculate the difference between the predicted output of the model and the true label to obtain the total loss value. Starting from the output layer, the gradient of the loss function is back-propagated using the chain rule. This involves calculating the gradient of the loss function to the output layer activation, and then reversely calculating the gradient of the loss to each layer weight and bias layer by layer. For each layer, the gradient of the loss function is calculated by the gradient of the previous layer in the network and the derivative of the activation function of the current layer. This process continues to the input layer. During the back propagation process, the chain rule is used to convert the gradient of the loss function to the output into the gradient of each parameter in the network.

[0101] Based on the calculated gradients, an optimization algorithm (such as gradient descent) is used to update the parameters of the model. Determine which optimization algorithm is used to update the model parameters. Common optimization algorithms include gradient descent, stochastic gradient descent (SGD), Adam, RMSprop, etc. According to the rules of the optimizer, the calculated gradients are used to update each model parameter. The learning rate is a hyperparameter that controls the step size of the parameter update. If the learning rate is too large, it may cause unstable training; if it is too small, the training process will be very slow. Some optimizers (such as SGD with Momentum) use past gradient information to accelerate convergence. Optimizers such as Adam and RMS prop adaptively adjust the learning rate based on the gradient history of each parameter. When updating parameters, weight decay (L2 regularization) is usually applied to prevent the model from overfitting. Each optimizer has its own parameter update rules. For example, the Adam optimizer maintains the first-order moment estimate and the second-order moment estimate of the gradient, and then uses these moment estimates to update the parameters.

[0102] During training, you may need to adjust the learning rate to speed up convergence or prevent oscillations based on the performance of the model. Apply regularization techniques (such as L1 or L2 regularization) to prevent the model from overfitting. Monitor the performance on the validation set and stop training if there is no improvement for multiple epochs in a row to avoid overfitting. Model training is performed iteratively. In each iteration, the model first performs forward propagation to calculate the output and loss, then calculates the gradient through backpropagation, and finally uses the optimization algorithm to update the parameters according to the gradient. This process is repeated until the performance of the model on the validation set is no longer significantly improved, or the preset number of iterations is reached.

[0103] In one embodiment, after the steps of calculating the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, updating the model parameters according to the total loss value and completing the speech conversion model training, the method further includes:

[0104] The encoder obtains the target speaker embedding based on the target speaker’s Mel-spectrogram.

[0105] The encoder obtains the source speaker content embedding and the source speaker latent code according to the source speaker's mel spectrum;

[0106] The source speaker content embedding, the source speaker hidden code and the target speaker embedding are input into the corresponding level decoder for information fusion and upsampling operations, and the characteristic Mel spectrum of the target speaker is output to complete the speech conversion.

[0107] In this embodiment, after the above training, the encoder, with the assistance of the classifier, can generate speaker embeddings that are independent of the speaker and speaker embeddings with good expressiveness, while the decoder can effectively fuse and upsample the output of the encoder to finally generate the target audio mel-spectrogram.

[0108] Further reference to Figure 3, as Figure 3 , which is a flow chart of an embodiment of a speech conversion method based on a speech conversion model of the present invention. The Mel spectrum of the target speaker is input into the encoder, and the encoder generates the target speaker embedding according to the Mel spectrum encoding of the target speaker. Among them, each module is responsible for extracting features at different levels, and each encoder module outputs a target speaker embedding.

[0109] Further reference to Figure 4, such as Figure 4 As shown, the Mel spectrum of the source speaker is input into the encoder, and the encoder generates the source speaker hidden code and the source speaker content embedding corresponding to the target speaker embedding according to the Mel spectrum coding of the source speaker.

[0110] The source speaker content embedding output by the encoder is fused with the target speaker embedding, and the fused information is fed into the decoder module at the corresponding level of the embedding input, and the latent code is decoded and upsampled step by step. When performing the decoding operation step by step, the decoder module at each level outputs the decoded latent code, which is then used as the input of the decoder at the next level. The output of the decoder module at the last layer is the mel spectrum converted into the target speaker's features to complete the speech conversion. In practical applications, the model may also include some other components, such as attention mechanisms, to enhance the model's ability to align the source and target speaker features. Model training usually involves a large amount of data and complex optimization algorithms to ensure that the model can accurately learn and simulate the conversion between speakers. The inference process may also include some post-processing steps, such as smoothing or denoising the speech signal, to improve the quality of the final output.

[0111] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0112] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, training technology for large speech conversion models, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0113] In an embodiment of the present invention, the encoder processes the input mel spectrum, outputs speaker embedding, content embedding and hidden code, and inputs the speaker embedding, content embedding and hidden code to the decoder; after receiving the speaker embedding, content embedding and hidden code, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations; the total loss value is calculated through the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum, and the model parameters are updated according to the total loss value. The present invention obtains the speaker embedding, content embedding and hidden code through the encoder, and the decoder reconstructs the mel spectrum. The model is trained by calculating the loss value according to the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum, which effectively avoids the damage and loss of audio information, improves the model's ability to capture and imitate the speaker's voice features, and the naturalness of voice conversion.

[0114] Further references Figure 5 , as a response to the above Figure 5 The present invention provides an embodiment of a training device for a speech conversion model. Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0115] like Figure 5 As shown, the training device of the speech conversion model described in this embodiment includes: an encoder module 11, a decoder module 12 and a model training module 13. Among them:

[0116] The encoder module 11 is configured to process the input mel spectrum by an encoder, output speaker embedding, content embedding and latent code, and input the speaker embedding, content embedding and latent code to a decoder;

[0117] A decoder module 12, configured to generate a reconstructed mel spectrum through feature fusion and upsampling operations after the decoder receives the speaker embedding, content embedding and latent coding;

[0118] The model training module 13 is used to calculate the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, update the model parameters according to the total loss value, and complete the speech conversion model training.

[0119] like Figure 5As shown, the encoder module 11, the decoder module 12 and the model training module 13 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of training and conversion of the speech conversion model than the program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0120] In one embodiment, the training device for the speech conversion model further includes:

[0121] The data processing module is used to obtain training speech data, pre-process the training speech data to obtain a training data set; and convert the training speech data set into a mel spectrum.

[0122] In one embodiment, the encoder module 11 specifically includes:

[0123] A hidden coding unit, used for the encoder to extract features from the mel spectrum to obtain hidden coding;

[0124] A content embedding unit, configured to perform a vector quantization operation on the latent code to generate a content embedding;

[0125] A speaker embedding unit, used to calculate the difference between the latent code and the content embedding to obtain the speaker embedding;

[0126] A data input unit is used to input the speaker embedding, content embedding and latent coding into a decoder.

[0127] In one embodiment, the decoder module 12 specifically includes:

[0128] A feature fusion unit, used for fusing the speaker embedding and the content embedding through the decoder to obtain a feature embedding;

[0129] The mel spectrum reconstruction unit is used to add the feature embedding to the latent code and perform upsampling processing to obtain a reconstructed mel spectrum.

[0130] In one embodiment, the model training module 13 specifically includes:

[0131] A reconstruction loss unit, used for calculating a reconstruction loss according to the Mel spectrum and the reconstructed Mel spectrum;

[0132] A classification loss unit, used for inputting the speaker embedding and the content embedding into a corresponding classifier to calculate the classification loss;

[0133] Quantization loss unit, used to calculate the quantization loss when performing vector quantization operation on the latent code;

[0134] The total loss unit is used to calculate the total loss value of reconstruction loss, classification loss and quantization loss by weighted summation.

[0135] In one embodiment, the model training module 13 specifically includes:

[0136] A gradient calculation unit, used to calculate the gradient of model parameters according to the total loss value using the back-propagation algorithm;

[0137] The model training unit is used to update the model parameters according to the gradient through the optimization algorithm to complete the speech conversion model training.

[0138] In one embodiment, a training device for a speech conversion model further includes a speech conversion module:

[0139] A target speaker unit, which is used by the encoder to obtain a target speaker embedding based on the target speaker's mel-spectrogram;

[0140] A source speaker unit, used for the encoder to obtain source speaker content embedding and source speaker latent coding according to the source speaker's mel spectrum;

[0141] The speech conversion unit is used to input the source speaker content embedding, the source speaker hidden code and the target speaker embedding into the corresponding level decoder for information fusion and upsampling operations, output the characteristic Mel spectrum of the target speaker, and complete the speech conversion.

[0142] In the above embodiment, the present invention discloses a training device for a speech conversion model, which belongs to the technical field of speech data processing. The present invention processes the input mel spectrum by an encoder, outputs speaker embedding, content embedding and hidden code, and inputs the speaker embedding, content embedding and hidden code to a decoder; after the decoder receives the speaker embedding, content embedding and hidden code, it generates a reconstructed mel spectrum through feature fusion and upsampling operations; the total loss value is calculated through the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum, and the model parameters are updated according to the total loss value to complete the speech conversion model training. The present invention obtains speaker embedding, content embedding and hidden code through an encoder, and the decoder reconstructs the mel spectrum. The loss value is calculated according to the target mel spectrum, speaker embedding, content embedding and reconstructed mel spectrum to train the model, effectively avoiding damage and loss of audio information, improving the model's ability to capture and imitate the speaker's voice features, and the naturalness of speech conversion.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0144] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0145] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0146] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 having components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0147] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0148] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of a training method for a speech conversion model, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.

[0149] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other training chips for speech conversion models in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the training method of the speech conversion model.

[0150] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0151] The computer device proposed in this embodiment belongs to the field of speech data processing technology and is applied to dialogue scenarios such as finance or medical treatment. The present invention processes the input Mel spectrum by an encoder, outputs speaker embedding, content embedding and hidden coding, and inputs the speaker embedding, content embedding and hidden coding to the decoder; after the decoder receives the speaker embedding, content embedding and hidden coding, it generates a reconstructed Mel spectrum through feature fusion and upsampling operations; the total loss value is calculated through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, and the model parameters are updated according to the total loss value. The present invention obtains speaker embedding, content embedding and hidden coding through an encoder, and the decoder reconstructs the Mel spectrum. The model is trained by calculating the loss value based on the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, which effectively avoids damage and loss of audio information, improves the model's ability to capture and imitate the speaker's voice features, and the naturalness of voice conversion.

[0152] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the training method of the speech conversion model as described above.

[0153] The computer-readable storage medium proposed in this embodiment belongs to the field of speech data processing technology and is applied to dialogue scenarios such as finance or medical treatment. The present invention processes the input Mel spectrum by an encoder, outputs speaker embedding, content embedding and hidden coding, and inputs the speaker embedding, content embedding and hidden coding to the decoder; after the decoder receives the speaker embedding, content embedding and hidden coding, it generates a reconstructed Mel spectrum through feature fusion and upsampling operations; calculates the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, and updates the model parameters according to the total loss value. The present invention obtains speaker embedding, content embedding and hidden coding through an encoder, and the decoder reconstructs the Mel spectrum. The loss value calculated according to the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum is used to train the model, effectively avoiding damage and loss of audio information, improving the model's ability to capture and imitate the speaker's voice features, and the naturalness of voice conversion.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0155] It should be noted that if software tools or components other than those of the company appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The embodiments described above are only some embodiments of the present invention, not all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the scope of the invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A method for training a speech conversion model, characterized in that: The steps include: The encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent code, and inputs the speaker embedding, content embedding and latent code to the decoder; After receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations; The total loss value is calculated through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, and the model parameters are updated according to the total loss value to complete the speech conversion model training.

2. The method for training a speech conversion model according to claim 1, characterized in that: The encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent coding, and inputs the speaker embedding, content embedding and latent coding to the decoder, including: Acquire training speech data, and preprocess the training speech data to obtain a training data set; The training speech data set is converted into mel-spectrogram.

3. The method for training a speech conversion model according to claim 1 or 2, characterized in that: The encoder processes the input mel spectrum, outputs speaker embedding, content embedding and latent coding, and inputs the speaker embedding, content embedding and latent coding to the decoder, including: The encoder extracts features from the mel spectrum to obtain latent codes; Performing a vector quantization operation on the latent code to generate a content embedding; Calculating the difference between the latent code and the content embedding to obtain speaker embedding; The speaker embedding, content embedding and latent coding are input to the decoder.

4. The method for training a speech conversion model according to claim 1, characterized in that: After receiving the speaker embedding, content embedding and latent coding, the decoder generates a reconstructed mel spectrum through feature fusion and upsampling operations, including: The decoder performs feature fusion on the speaker embedding and the content embedding to obtain a feature embedding; The feature embedding is added to the latent code, and up-sampling is performed to obtain a reconstructed Mel spectrum.

5. The method for training a speech conversion model according to claim 1, characterized in that: The total loss value is calculated by using the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, including: Calculating a reconstruction loss according to the Mel spectrum and the reconstructed Mel spectrum; Input the speaker embedding and content embedding into the corresponding classifier to calculate the classification loss; Calculate the quantization loss when performing vector quantization operations on the hidden code; The total loss value is calculated by weighted summation of reconstruction loss, classification loss, and quantization loss.

6. The method for training a speech conversion model according to claim 5, characterized in that: The updating of the model parameters according to the total loss value to complete the speech conversion model training includes: Use the back-propagation algorithm to calculate the gradient of the model parameters based on the total loss value; The model parameters are updated according to the gradient through the optimization algorithm to complete the speech conversion model training.

7. The method for training a speech conversion model according to claim 1, characterized in that: After the steps of calculating the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, updating the model parameters according to the total loss value, and completing the speech conversion model training, the method further includes: The encoder obtains the target speaker embedding based on the target speaker’s Mel-spectrogram. The encoder obtains the source speaker content embedding and the source speaker latent code according to the source speaker's mel spectrum; The source speaker content embedding, the source speaker hidden code and the target speaker embedding are input into the corresponding level decoder for information fusion and upsampling operations, and the characteristic Mel spectrum of the target speaker is output.

8. A training device for a speech conversion model, characterized in that: include: An encoder module, configured to process the input mel spectrum by an encoder, output speaker embedding, content embedding and latent code, and input the speaker embedding, content embedding and latent code to a decoder; A decoder module, configured to generate a reconstructed mel spectrum through feature fusion and upsampling operations after the decoder receives the speaker embedding, content embedding and latent coding; The model training module is used to calculate the total loss value through the target Mel spectrum, speaker embedding, content embedding and reconstructed Mel spectrum, update the model parameters according to the total loss value, and complete the speech conversion model training.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for training a speech conversion model as described in any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for training a speech conversion model according to any one of claims 1 to 7.

Images