End-to-end voice conversion method, system, terminal and storage medium

Abstract

The present invention discloses an end-to-end speech conversion method, system, terminal and storage medium. The method comprises: obtaining source speech data of a source speaker and a mel spectrum of the source speech data; inputting the source speech data and the mel spectrum into a pre-trained speech conversion model, calculating a first latent variable of the source speech data through the speech conversion model, the first latent variable containing speech content information in the source speech data and timbre information of the source speaker; separating a second latent variable containing only speech content information from the source speech data based on the first latent variable; performing speech conversion according to the second latent variable and the target speaker to obtain the converted speech data of the target speaker. The present invention does not need to rely on parallel data sets, saving a lot of data set production costs. At the same time, it does not need to rely on a vocoder, avoiding the problem of direct feature mismatch between the speech conversion model and the vocoder, and improving the speech conversion quality.

Description

Technical Field

[0001] The present invention relates to the technical field of voice conversion, and in particular to an end-to-end voice conversion method, system, terminal and storage medium. Background Art

[0002] Speech conversion has received extensive attention in many applications of speech synthesis, typical applications include movie dubbing, pronunciation assistance, etc. The goal of speech conversion is to convert the timbre of the source speaker to the timbre of the target speaker while keeping the language content unchanged. Traditional speech conversion methods require the collection of a large amount of parallel training data. These parallel training data require different speakers to record the same language content and perform operations such as time alignment and manual correction, which is time-consuming and difficult to implement in some cases. The speech conversion method based on GAN (Generative Adversarial Networks) learns the mapping relationship between data through non-parallel training data to perform speech conversion, but its speech conversion effect is still difficult to meet the needs. In addition, traditional speech conversion can only convert acoustic features, and the vocoder needs to be used to restore the acoustic features to speech. However, there will be noise caused by the mismatch of acoustic features in the vocoder training and inference stages, especially under the condition of limited training data. Summary of the invention

[0003] The present invention provides an end-to-end speech conversion method, system, terminal and storage medium, aiming to solve the technical problems such as the time-consuming need of existing speech conversion methods to rely on parallel training data and the noise caused by the acoustic feature mismatch problem of the need to rely on the vocoder.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] An end-to-end voice conversion method, comprising:

[0006] Acquire source speech data of a source speaker and a mel spectrum of the source speech data;

[0007] Inputting the source speech data and the mel spectrum into a pre-trained speech conversion model, and calculating a first latent variable of the source speech data through the speech conversion model, wherein the first latent variable includes speech content information in the source speech data and timbre information of the source speaker;

[0008] Separating a second latent variable containing only speech content information from the source speech data based on the first latent variable;

[0009] Speech conversion is performed according to the second latent variable and the target speaker to obtain converted speech data of the target speaker.

[0010] The technical solution adopted by the embodiment of the present invention also includes: the obtaining of source speech data of a source speaker and the mel spectrum of the source speech data includes:

[0011] A Mel spectrum is extracted from the source speech data using a Mel filter bank.

[0012] The technical solution adopted by the embodiment of the present invention also includes: extracting the mel spectrum from the source speech data using the mel filter bank includes:

[0013] Calculating the amplitude spectrum of the source speech data using a fast Fourier transform algorithm;

[0014] The amplitude spectrum is converted by a Mel filter bank of a set dimension to obtain a Mel spectrum of the source speech data.

[0015] The technical solution adopted by the embodiment of the present invention also includes: the speech conversion model is based on a conditional transformation autoencoder structure, the speech conversion model includes an encoder, a decoder and a discriminator, the encoder and decoder respectively include a residual layer, a downsampling layer and an upsampling layer, the residual layer is used to calculate and extract hidden states from the input, and the speaker embedding is used as a conditional input for modeling different speakers; the downsampling layer and upsampling are used to downsample and upsample the input respectively; the discriminator is used to distinguish the output generated by the decoder from the real data sample, and perform adversarial training on the speech conversion model.

[0016] The technical solution adopted by the embodiment of the present invention also includes: the first latent variable of the source speech data calculated by the speech conversion model includes:

[0017] The encoder calculates the first latent variable z containing the speech content information in the source speech data and the timbre information of the source speaker s according to the input source speech data and the mel spectrum x; the first latent variable z obeys the conditional probability distribution based on the source speaker s and the mel spectrum x, and the calculation formula of the first latent variable z is:

[0018] z~q φ (z|x,s)

[0019] The technical solution adopted by the embodiment of the present invention also includes: the second latent variable containing only voice content information is separated from the source voice data based on the first latent variable, including:

[0020] Based on the first latent variable z, the decoder normalizes the flow function f θSeparating the speech content information and the timbre information of the source speaker s in the source speech data to obtain a second latent variable c containing only the speech content information and a third latent variable e for representing the timbre information of the source speaker s;

[0021] The calculation formula of the second latent variable c is:

[0022] c=f θ (z|s)

[0023] The technical solution adopted by the embodiment of the present invention also includes: performing voice conversion according to the second latent variable and the target speaker to obtain the converted voice data of the target speaker includes:

[0024] Based on the second latent variable c and the target speaker Using the normalizing flow function f θ Inverse transformation Calculate the fourth latent variable after timbre conversion The fourth hidden variable Contains the same speech content information as the first latent variable z, and includes the target speaker The timbre information;

[0025] The fourth latent variable after the timbre conversion Generate the target speaker after voice conversion through the decoder network Voice data

[0026]

[0027] Another technical solution adopted by the embodiment of the present invention is: an end-to-end voice conversion system, comprising:

[0028] Data acquisition module: used to acquire source speech data of a source speaker and Mel spectrum of the source speech data;

[0029] A first extraction module is used to input the source speech data and the mel spectrum into a pre-trained speech conversion model, and calculate the first latent variable of the source speech data through the speech conversion model, wherein the first latent variable includes speech content information in the source speech data and timbre information of the source speaker;

[0030] A second extraction module: used for separating a second latent variable containing only voice content information from the source voice data based on the first latent variable;

[0031] Speech conversion module: used to perform speech conversion according to the second latent variable and the target speaker to obtain the converted speech data of the target speaker.

[0032] Another technical solution adopted by an embodiment of the present invention is: a terminal, the terminal includes a processor and a memory coupled to the processor, wherein:

[0033] The memory stores program instructions for implementing the above-mentioned end-to-end voice conversion method;

[0034] The processor is configured to execute the program instructions stored in the memory to perform the end-to-end voice conversion operation.

[0035] Another technical solution adopted by the embodiment of the present invention is: a storage medium stores program instructions executable by a processor, and the program instructions are used to execute the above-mentioned end-to-end voice conversion method.

[0036] The end-to-end speech conversion method, system, terminal and storage medium of the embodiment of the present invention are based on the conditional transformation autoencoder structure, which separates the timbre information of the source speaker from the speech content, and performs speech conversion of the target speaker according to the speech content after timbre separation. Compared with the existing speech conversion methods, the present invention does not need to rely on parallel data sets, saving a lot of data set production costs. At the same time, it directly generates speech information instead of acoustic features, without relying on the vocoder, avoiding the problem of mismatch between the speech conversion model and the direct features of the vocoder, making the converted speech more natural and similar, and improving the quality of speech conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of an end-to-end voice conversion method according to a first embodiment of the present invention;

[0038] Figure 2 is a flow chart of an end-to-end voice conversion method according to a second embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of the encoder and decoder network structure of an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of the Residual Stack structure of an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of the structure of an end-to-end voice conversion system according to an embodiment of the present invention;

[0042] Figure 6 is a schematic diagram of the terminal structure of an embodiment of the present invention;

[0043] Figure 7 It is a schematic diagram of the storage medium structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The terms "first", "second" and "third" in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0046] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] See also Figure 1 , is a flow chart of an end-to-end voice conversion method according to a first embodiment of the present invention. The end-to-end voice conversion method according to the first embodiment of the present invention comprises the following steps:

[0048] S100: Acquire source speech data of a source speaker and a mel spectrum of the source speech data;

[0049] S101: inputting the source speech data and the mel spectrum into a pre-trained speech conversion model, and calculating the first latent variable of the source speech data through the speech conversion model, where the first latent variable includes speech content information in the source speech data and timbre information of the source speaker;

[0050] S102: Separating a second latent variable containing only speech content information from the source speech data based on the first latent variable;

[0051] S103: Perform voice conversion according to the second latent variable and the target speaker to obtain converted voice data of the target speaker.

[0052] See also Figure 2 , is a flow chart of an end-to-end voice conversion method according to a second embodiment of the present invention. The end-to-end voice conversion method according to the second embodiment of the present invention comprises the following steps:

[0053] S200: Acquire source speech data of a source speaker;

[0054] S201: extracting a mel spectrum from the source speech data using a mel filter bank;

[0055] In this step, the Mel spectrum extraction process of the source speech data is specifically as follows: the amplitude spectrum of the source speech data is calculated using the FFT (fast Fourier transform) algorithm, and the amplitude spectrum is converted through a Mel filter group of a certain dimension, and the result is the Mel spectrum x of the source speech data. Among them, FFT is an efficient algorithm of DFT (Discrete Fourier Transform), called fast Fourier transform. The amplitude spectrum characterizes the distribution of the amplitude of the signal with frequency. After the source speech data is transformed into the frequency domain using the FFT algorithm, the amplitude spectrum can be obtained by performing a modulo operation on the source speech data.

[0056] S201: Input the source speech data and the Mel spectrum x into the pre-trained speech conversion model, and the speech conversion model calculates the first latent variable z of the source speech data through the encoder, wherein the first latent variable z contains the speech content information in the source speech data and the timbre information of the source speaker s;

[0057] In this step, the speech conversion model is based on the CVAE (conditional variational autoencoder) structure. The CVAE structure includes two parts: encoder and decoder. Each time the encoder and decoder are input, a label is input. During machine learning training, it can be converted into a vector of a hot word. Even if two points are in the same position, different generated data can be obtained according to different labels. In the speech conversion model based on the CVAE structure, the encoder is used to calculate the first latent variable z containing the speech content information in the source speech data and the timbre information of the source speaker s based on the input source speech data and the Mel spectrum x. The first latent variable z obeys the conditional probability distribution based on the source speaker s and the Mel spectrum x. The calculation formula of the first latent variable z is:

[0058] z~q φ (z|x,s) (1)

[0059] The decoder is used to normalize the first latent variable z through the normalizing flow function f θ The speech content information and the timbre information of the source speaker s are separated from the source speech data, thereby obtaining the second latent variable c containing only the speech content information and the third latent variable e used to characterize the timbre information of the source speaker s. The second latent variable c and the target speaker By normalizing the flow function f θ Inverse transformation And decoder can generate voice data after timbre conversion

[0060] In the embodiment of the present invention, the network structure of the encoder and the decoder is as follows: Figure 3 As shown in the figure, it consists of four layers: ResidualStack, Dowansample and Upsample. The ResidualStack layer is used to calculate and extract hidden states from the input to represent the input features; Dowansample and Upsample are used to downsample and upsample the input respectively to reduce or increase the resolution. Figure 4 As shown in the figure, speaker embedding is added as a conditional input to distinguish the calculation process of different speakers and model different speakers.

[0061] In the embodiment of the present invention, the optimization goal of the speech conversion model based on the CVAE structure is to maximize the ELBO (Evidence Lower Bound), that is, to minimize the recovery loss L recon =-logp θ (x / z) and KL (relative entropy) divergence L kl =logq φ (z|x)-logp θ When training the speech conversion model, the Mel spectrum feature is used Calculate the recovery loss and generate speech data by upsampling the first latent variable z through the decoder From the voice data Extracting Mel spectrum features The restoration loss for the voice conversion model is defined as The first latent variable z follows a Gaussian distribution z~q φ (z|x)=N(z;u φ (x),σ φ (x)), where N(z;u φ (x),σ φ (x)) represents the φ (x),σ φ (x) is a Gaussian distribution with mean and standard deviation. θ (z|c) probability distribution using normalizing flow function f θ Modeling, then Indicates u θ (c), σ θ (c) is the Gaussian distribution with mean and standard deviation, is the determinant. The normalizing flow function f θ A series of reversible mappings are used to transform the original distribution into a new distribution. By optimizing this series of distributions, the purpose of transforming a simple Gaussian distribution into a complex true posterior distribution can be achieved.

[0062] In the embodiment of the present application, in order to achieve adversarial training, the discriminator is added to distinguish the output generated by the decoder. Compared with the real data sample y, the adversarial training loss function is defined as:

[0063] L adv (D) = (D(y) - 1) 2 +D(G(z)) 2 (2)

[0064] L adv (G) = (D (G (2)) - 1) 2 (3)

[0065] Among them, G represents the decoder and D represents the discriminator.

[0066] Combining ELBO and adversarial training, the loss function of the speech conversion model training is L total =L recon +L kl +L adv (G), the loss function of the discriminator D training is L adv (D).

[0067] Based on the above, the speech conversion model of the embodiment of the present invention adopts a fully convolutional structure, which has a higher inference efficiency than the existing autoregressive network structure. At the same time, it also has the potential to be expanded to zero-shot speech conversion.

[0068] S202: Based on the first latent variable z, the decoder uses the normalizing flow function f θ Separate the speech content information and the timbre information of the source speaker s, and obtain a second latent variable c containing only the speech content information and a third latent variable e used to represent the timbre information of the source speaker;

[0069] In this step, the first latent variable z is normalized by the normalizing flow function f under the condition that the source speaker s is known. θ The second latent variable c is calculated to be independent of the source speaker and contains only speech content information:

[0070] c=f θ (z|s) (4)

[0071] S203: Based on the second latent variable c and the target speaker By normalizing the flow function f θ Inverse transformation After that, the converted target speaker is obtained through the decoder Voice data;

[0072] In this step, based on the second latent variable c and the target speaker First, use the inverse transformation of the normalizing flow function Calculate the fourth latent variable after timbre conversion The fourth hidden variable Contains the same speech content information as the first latent variable z, and includes the target speaker The timbre information of the timbre is converted into the fourth latent variable Through the decoder network, voice data after timbre conversion can be generated

[0073]

[0074] Based on the above, the end-to-end speech conversion method of the second embodiment of the present invention is based on the conditional transformation autoencoder structure. Through adversarial training, the timbre information of the source speaker is separated from the speech content, and the speech conversion of the target speaker is performed according to the speech content after timbre separation. Compared with the existing speech conversion methods, the present invention does not need to rely on parallel data sets, saving a lot of data set production costs. At the same time, it directly generates speech information instead of acoustic features, without relying on the vocoder, avoiding the problem of mismatch between the speech conversion model and the direct features of the vocoder, making the converted speech more natural and similar, and improving the quality of speech conversion.

[0075] In an optional implementation, it is also possible to upload the result of the end-to-end voice conversion method to the blockchain.

[0076] Specifically, the corresponding summary information is obtained based on the result of the end-to-end voice conversion method. Specifically, the summary information is obtained by hashing the result of the end-to-end voice conversion method, such as by using the sha256s algorithm. Uploading the summary information to the blockchain can ensure its security and fairness and transparency to users. Users can download the summary information from the blockchain to verify whether the result of the end-to-end voice conversion method has been tampered with. The blockchain referred to in this example is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0077] See also Figure 5 , is a schematic diagram of the structure of an end-to-end voice conversion system according to an embodiment of the present invention. The end-to-end voice conversion system 40 according to an embodiment of the present invention comprises:

[0078] Data acquisition module 41: used to obtain the source speech data of the source speaker and the Mel spectrum of the source speech data; wherein, the Mel spectrum extraction process of the source speech data by the data acquisition module 41 is specifically as follows: using the FFT (fast Fourier transform) algorithm to calculate the amplitude spectrum of the source speech data, and transforming the amplitude spectrum through a Mel filter group of a certain dimension, and the result obtained is the Mel spectrum x. Among them, FFT is an efficient algorithm of DFT (Discrete Fourier Transform), called fast Fourier transform, and the amplitude spectrum represents the distribution of the amplitude of the signal with frequency. After the source speech data is transformed into the frequency domain using the FFT algorithm, the amplitude spectrum can be obtained by performing a modulus operation on the source speech data.

[0079] The first extraction module 42 is used to input the source speech data and the Mel spectrum x into the pre-trained speech conversion model. The speech conversion model calculates the first latent variable z of the source speech data through the encoder, wherein the first latent variable z contains the speech content information in the source speech data and the timbre information of the source speaker s; wherein the speech conversion model is based on the CVAE (conditional variational autoencoder) structure, and the CVAE structure includes two parts, the encoder and the decoder. Each time the encoder and the decoder are input, a label is input, which can be converted into a vector of a hot word during machine learning training. Even if two points are in the same position, different generated data can be obtained according to different labels. In the speech conversion model based on the CVAE structure, the encoder is used to calculate the first latent variable z containing the speech content information in the source speech data and the timbre information of the source speaker s according to the input source speech data and the Mel spectrum x. The first latent variable z obeys the conditional probability distribution based on the source speaker s and the Mel spectrum x, and the calculation formula of the first latent variable z is:

[0080] z~q φ (z|x,s) (1)

[0081] The decoder is used to normalize the first latent variable z through the normalizing flow function f θ The speech content information and the timbre information of the source speaker s are separated from the source speech data, thereby obtaining the second latent variable c containing only the speech content information and the third latent variable e used to characterize the timbre information of the source speaker s. The second latent variable c and the target speaker By normalizing the flow function f θ Inverse transformation And decoder can generate voice data after timbre conversion

[0082] In the embodiment of the present invention, the network structure of the encoder and the decoder is as follows: Figure 3 As shown in the figure, it consists of four layers: ResidualStack, Dowansample and Upsample. The ResidualStack layer is used to calculate and extract hidden states from the input to represent the input features; Dowansample and Upsample are used to downsample and upsample the input respectively to reduce or increase the resolution. Figure 4 As shown in the figure, speaker embedding is added as a conditional input to distinguish the calculation process of different speakers and model different speakers.

[0083] In the embodiment of the present invention, the optimization goal of the speech conversion model based on the CVAE structure is to maximize the ELBO (Evidence Lower Bound), that is, to minimize the recovery loss L recon =-logp θ (x / z) and KL (relative entropy) divergence L kl =logq φ (z|x)-logp θ When training the speech conversion model, the Mel spectrum feature is used Calculate the recovery loss and generate speech data by upsampling the first latent variable z through the decoder From the voice data Extracting Mel spectrum features The restoration loss for the voice conversion model is defined as The first latent variable z follows a Gaussian distribution z~q φ (z|x)=N(z;u φ (x),σ φ (x)), where N(z;u φ (x),σ φ (x)) represents the φ (x),σ φ (x) is a Gaussian distribution with mean and standard deviation. θ (z|c) probability distribution using normalizing flow function f θ Modeling, then Indicates uθ (c), σ θ (c) is the Gaussian distribution with mean and standard deviation, is the determinant. The normalizing flow function f θ A series of reversible mappings are used to transform the original distribution into a new distribution. By optimizing this series of distributions, the purpose of transforming a simple Gaussian distribution into a complex true posterior distribution can be achieved.

[0084] In the embodiment of the present application, in order to achieve adversarial training, the discriminator is added to distinguish the output generated by the decoder. Compared with the real data sample y, the adversarial training loss function is defined as:

[0085] L adv (D) = (D(y) - 1) 2 +D(G(z)) 2 (2)

[0086] L adv (G) = (D (G (2)) - 1) 2 (3)

[0087] Among them, G represents the decoder and D represents the discriminator.

[0088] Combining ELBO and adversarial training, the loss function of the speech conversion model training is L total =L recon +L kl +L adv (G), the loss function of the discriminator D training is L adv (D).

[0089] Based on the above, the speech conversion model of the embodiment of the present invention adopts a fully convolutional structure, which has a higher inference efficiency than the existing autoregressive network structure. At the same time, it also has the potential to be expanded to zero-shot speech conversion.

[0090] The second extraction module 43 is used for the decoder to decode the first latent variable z through the normalizing flow function f θ Separate the speech content information and the timbre information of the source speaker s, and obtain a second latent variable c containing only the speech content information and a third latent variable e used to represent the timbre information of the source speaker;

[0091] In this step, the first latent variable z is normalized by the normalizing flow function f under the condition that the source speaker s is known. θ The second latent variable c is calculated to be independent of the source speaker and contains only speech content information:

[0092] c=f θ (z|s) (4)

[0093] Voice conversion module 44: used to convert the target speaker into a voice based on the second latent variable c By normalizing the flow function f θ Inverse transformation and decoder to get the converted target speaker wherein the speech conversion process of the speech conversion module 45 is specifically as follows: based on the second latent variable c and the target speaker First, use the inverse transformation of the normalizingflow function Calculate the fourth latent variable after timbre conversion The fourth hidden variable Contains the same speech content information as the first latent variable z, and includes the target speaker The timbre information of the timbre is converted into the fourth latent variable Through the decoder network, voice data after timbre conversion can be generated

[0094]

[0095] Based on the above, the end-to-end speech conversion system of the embodiment of the present invention is based on the conditional transformation autoencoder structure. Through adversarial training, the timbre information of the source speaker is separated from the speech content, and the speech conversion of the target speaker is performed according to the speech content after timbre separation. Compared with the existing speech conversion methods, the present invention does not need to rely on parallel data sets, saving a lot of data set production costs. At the same time, it directly generates speech information instead of acoustic features, without relying on the vocoder, avoiding the problem of mismatch between the speech conversion model and the direct features of the vocoder, making the converted speech more natural and similar, and improving the quality of speech conversion.

[0096] See also Figure 6 , is a schematic diagram of a terminal structure of an embodiment of the present invention. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

[0097] The memory 52 stores program instructions for implementing the above end-to-end voice conversion method.

[0098] The processor 51 is used to execute program instructions stored in the memory 52 to perform end-to-end voice conversion operations.

[0099] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0100] See also Figure 7 , Figure 7 Schematic diagram of the structure of the storage medium of the embodiment of the present invention. The storage medium of the embodiment of the present invention stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0101] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present invention.

Claims

1. An end-to-end voice conversion method, characterized in that: include: Acquire source speech data of a source speaker and a mel spectrum of the source speech data; Inputting the source speech data and the mel spectrum into a pre-trained speech conversion model, and calculating a first latent variable of the source speech data through the speech conversion model, wherein the first latent variable includes speech content information in the source speech data and timbre information of the source speaker; The speech conversion model is based on a conditional transform autoencoder structure, and includes an encoder, a decoder, and a discriminator. The encoder and the decoder include a residual layer, a downsampling layer, and an upsampling layer, respectively. The residual layer is used to calculate and extract hidden states from the input, and uses speaker embedding as a conditional input to model different speakers; the downsampling layer and the upsampling layer are used to downsample and upsample the input, respectively; The discriminator is used to distinguish the output generated by the decoder from the real data sample, and perform adversarial training on the speech conversion model; The first latent variable of the source speech data calculated by the speech conversion model includes calculating the first latent variable z containing speech content information in the source speech data and timbre information of the source speaker s according to the input source speech data and the mel spectrum x by the encoder; the first latent variable z obeys the conditional probability distribution based on the source speaker s and the mel spectrum x, and the calculation formula of the first latent variable z is: z~q φ (z|x,s); Separating a second latent variable containing only speech content information from the source speech data based on the first latent variable; said separating a second latent variable containing only speech content information from the source speech data based on the first latent variable comprises: based on the first latent variable z, the decoder uses a normalizing flow function f θ The speech content information and the timbre information of the source speaker s in the source speech data are separated to obtain a second latent variable c containing only the speech content information and a third latent variable e for representing the timbre information of the source speaker s. The calculation formula of the second latent variable c is: c=f θ (z|s); According to the second latent variable and the target speaker, voice conversion is performed to obtain the voice data of the target speaker after conversion; the voice conversion according to the second latent variable and the target speaker to obtain the voice data of the target speaker after conversion includes the following steps: Using the normalizing flow function f θ Inverse transformation Calculate the fourth latent variable after timbre conversion The fourth hidden variable Contains the same speech content information as the first latent variable z, and includes the target speaker The timbre information of the timbre is converted into a fourth latent variable Generate the target speaker after voice conversion through the decoder network Voice data G represents decoder.

2. The end-to-end voice conversion method according to claim 1, characterized in that: The acquiring source speech data of a source speaker and the mel spectrum of the source speech data comprises: A Mel spectrum is extracted from the source speech data using a Mel filter bank.

3. The end-to-end voice conversion method according to claim 2, characterized in that: The extracting the Mel spectrum from the source speech data using the Mel filter bank comprises: Calculating the amplitude spectrum of the source speech data using a fast Fourier transform algorithm; The amplitude spectrum is converted by a Mel filter bank of a set dimension to obtain a Mel spectrum of the source speech data.

4. An end-to-end voice conversion system, characterized in that: include: Data acquisition module: used to acquire source speech data of a source speaker and Mel spectrum of the source speech data; A first extraction module is used to input the source speech data and the mel spectrum into a pre-trained speech conversion model, and calculate the first latent variable of the source speech data through the speech conversion model, wherein the first latent variable includes speech content information in the source speech data and timbre information of the source speaker; The speech conversion model is based on a conditional transform autoencoder structure, and includes an encoder, a decoder, and a discriminator. The encoder and the decoder include a residual layer, a downsampling layer, and an upsampling layer, respectively. The residual layer is used to calculate and extract hidden states from the input, and uses speaker embedding as a conditional input to model different speakers; the downsampling layer and the upsampling layer are used to downsample and upsample the input, respectively; The discriminator is used to distinguish the output generated by the decoder from the real data sample, and perform adversarial training on the speech conversion model; The first latent variable of the source speech data calculated by the speech conversion model includes calculating the first latent variable z containing speech content information in the source speech data and timbre information of the source speaker s according to the input source speech data and the mel spectrum x by the encoder; the first latent variable z obeys the conditional probability distribution based on the source speaker s and the mel spectrum x, and the calculation formula of the first latent variable z is: z~q φ (z|x, s); The second extraction module is used to separate the second latent variable containing only the speech content information from the source speech data based on the first latent variable; the second latent variable containing only the speech content information from the source speech data based on the first latent variable includes: based on the first latent variable z, the decoder uses the normalizing flow function f θ The speech content information and the timbre information of the source speaker s in the source speech data are separated to obtain a second latent variable c containing only the speech content information and a third latent variable e for representing the timbre information of the source speaker s. The calculation formula of the second latent variable c is: c=f θ (z|s); A voice conversion module: used to perform voice conversion according to the second latent variable and the target speaker to obtain the converted voice data of the target speaker; the voice conversion according to the second latent variable and the target speaker to obtain the converted voice data of the target speaker includes the following steps: Using the normalizing flow function f θ Inverse transformation Calculate the fourth latent variable after timbre conversion The fourth hidden variable Contains the same speech content information as the first latent variable z, and includes the target speaker The timbre information of the timbre is converted into a fourth latent variable Generate the target speaker after voice conversion through the decoder network Voice data G represents decoder.

5. A terminal, characterized in that: The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the end-to-end voice conversion method according to any one of claims 1 to 3; The processor is configured to execute the program instructions stored in the memory to perform the end-to-end voice conversion method.

6. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the end-to-end voice conversion method according to any one of claims 1 to 3.

Images