Voice conversion method, training method, device, equipment and medium

Abstract

The voice conversion method, the training method, the device, the equipment and the medium provided by the embodiments of the present application obtain the source linear spectrum of the source speaker according to the source voice data of the source speaker; input the source linear spectrum into a pre-trained voice coding model to output corresponding spectral feature prediction data; input the spectral feature prediction data and target speaker feature data of a target speaker into a voice reconstruction model to output corresponding target voice data; in the above manner, the content information of the source voice data and the speaker feature are decoupled, and in the training stage and the voice conversion stage of the voice coding model, the input and the output of the voice coding model respectively only contain content information, the voice coding model is used to reconstruct voice data together with the content information and the speaker feature through the voice reconstruction model, which is beneficial to improve the training speed and the training effect of the voice coding model, and further improves the voice conversion effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a speech conversion method, a training method, a device, equipment and a medium. BACKGROUND

[0002] Speech conversion refers to a task of converting a voice of a source speaker into a timbre of a target speaker while preserving the speaking content of the source speaker. With the development of Internet technology, speech conversion has been gradually applied to the fields of electronic games, video live streaming, short video applications, etc.

[0003] Most of the speech conversion in the prior art adopts the following manner: obtaining content information of a speaker in a source speaker, the content information including phoneme information and / or spectrum information; training a generation model, and synthesizing new spectrum (e.g., mel spectrum or linear spectrum) by using the trained generation model to combine the content information and target speaker features; and converting the new spectrum into speech. Since the above manner trains the generation model by taking the content information and speaker features as inputs, it is not conducive to improving the training speed and training effect of the model, and thus is not conducive to improving the effect of speech conversion. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a speech conversion method, a training method, a device, equipment and a medium to solve the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide a speech conversion method, comprising:

[0006] obtaining a source linear spectrum of a source speaker from source speech data of the source speaker;

[0007] inputting the source linear spectrum into a pre-trained speech coding model to output corresponding spectrum feature prediction data, wherein the speech coding model is obtained by training according to sample phoneme sequences and sample linear spectrums extracted from sample speech data;

[0008] inputting the spectrum feature prediction data and target speaker feature data of a target speaker into a speech reconstruction model to output corresponding target speech data, wherein the speech reconstruction model is obtained by training according to sample spectrum feature data, sample speaker feature data and sample speech data, and the sample spectrum feature data is obtained from sample linear spectrums extracted from the sample speech data.

[0009] As an implementation manner, the training step of the speech coding model comprises:

[0010] obtaining at least one training sample, the training sample comprising a sample phoneme sequence and a sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained according to sample speech data;

[0011] inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence;

[0012] inputting the sample linear spectrum into the speech coding model to output sample spectrum feature prediction data of the sample linear spectrum;

[0013] obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectrum feature prediction data;

[0014] obtaining an index mapping vector of the sample spectrum feature prediction data according to the first alignment matrix and the phoneme feature data;

[0015] obtaining spectrum feature calculation data according to the index mapping vector;

[0016] obtaining a KL divergence of the spectrum feature calculation data and the sample spectrum feature prediction data, adjusting parameters of the speech coding model according to the KL divergence until the speech coding model reaches a training convergence condition.

[0017] As an implementation form, the spectrum feature calculation data is obtained according to the index mapping vector, comprising:

[0018] obtaining a second alignment matrix of the sample spectrum feature prediction data according to the index mapping vector and the phoneme feature data;

[0019] obtaining alignment position calculation data of the phoneme feature data according to the second alignment matrix and the sample spectrum feature prediction data;

[0020] obtaining a reconstruction alignment matrix according to the alignment position calculation data and the sample spectrum feature prediction data;

[0021] obtaining spectrum feature calculation data according to the reconstruction alignment matrix.

[0022] As an implementation form, the index mapping vector of the sample spectrum feature prediction data is obtained according to the first alignment matrix and the phoneme feature data, comprising:

[0023] obtaining a corresponding phoneme index vector according to the phoneme feature data;

[0024] obtaining the index mapping vector according to the first alignment matrix and the phoneme index vector;

[0025] Accordingly, the obtaining of the spectrum feature calculation data according to the index mapping vector comprises:

[0026] obtaining a reconstruction alignment matrix according to the index mapping vector and the phoneme index vector;

[0027] obtaining spectrum feature calculation data according to the reconstruction alignment matrix.

[0028] As an implementation form, the training step of the speech reconstruction model comprises:

[0029] obtaining at least one training sample, wherein the training sample comprises sample speech data and sample linear spectrum and sample speaker feature data extracted from the sample speech data;

[0030] inputting the sample linear spectrum into a speech coding model to output sample spectrum feature prediction data of the sample linear spectrum;

[0031] inputting the sample spectrum feature prediction data and the sample speaker feature data into a speech reconstruction model to output reconstructed sample speech data;

[0032] calculating a reconstruction loss according to the sample speech data and the reconstructed sample speech data, and adjusting parameters of the speech reconstruction model according to the reconstruction loss until the speech reconstruction model reaches a training convergence condition.

[0033] As an implementation form, the inputting of the spectrum feature prediction data and target speaker feature data of a target speaker into the speech reconstruction model to output corresponding target speech data comprises:

[0034] splicing the spectrum feature prediction data and the target speaker feature data of the target speaker to obtain spliced feature data;

[0035] obtaining the corresponding target speech data according to the spliced feature data.

[0036] In a second aspect, the embodiments of the present application provide a training method of a speech conversion model, wherein the speech conversion model comprises a speech coding model and a speech reconstruction model, and the training method comprises:

[0037] obtaining at least one training sample, wherein the training sample comprises sample speech data and sample linear spectrum, sample phoneme sequence and sample speaker feature data extracted from the sample speech data;

[0038] inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence;

[0039] input the sample linear spectrum into the speech coding model, and output sample spectrum feature prediction data of the sample linear spectrum;

[0040] obtain a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectrum feature prediction data;

[0041] obtain an index mapping vector of the sample spectrum feature prediction data according to the first alignment matrix and the phoneme feature data;

[0042] obtain spectrum feature calculation data according to the index mapping vector;

[0043] obtain a KL divergence of the spectrum feature calculation data and the sample spectrum feature prediction data, adjust parameters of the speech coding model according to the KL divergence, and stop until the speech coding model reaches a training convergence condition;

[0044] input the sample spectrum feature prediction data and the sample speaker feature data into a speech reconstruction model, and output reconstructed sample speech data;

[0045] calculate a reconstruction loss according to the sample speech data and the reconstructed sample speech data, adjust parameters of the speech reconstruction model according to the reconstruction loss, and stop until the speech reconstruction model reaches a training convergence condition.

[0046] In a third aspect, an embodiment of the present application provides a speech conversion device, comprising:

[0047] extract a source linear spectrum of a source speaker according to source speech data of the source speaker;

[0048] input the source linear spectrum into a pre-trained speech coding model, and output corresponding spectrum feature prediction data, wherein the speech coding model is trained according to a sample phoneme sequence and a sample linear spectrum obtained from sample speech data;

[0049] input the spectrum feature prediction data and target speaker feature data of a target speaker into a speech reconstruction model, and output corresponding target speech data, wherein the speech reconstruction model is trained according to sample spectrum feature data, sample speaker feature data and sample speech data, and the sample spectrum feature data is obtained according to a sample linear spectrum extracted from the sample speech data.

[0050] In a fourth aspect, an electronic device is provided, which includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; the processor executes the program instructions stored in the memory to implement the speech conversion method or the training method of the speech conversion model.

[0051] In a fifth aspect, a storage medium is provided, which stores program instructions, the program instructions being executed by a processor to implement the speech conversion method or the training method of the speech conversion model.

[0052] The speech conversion method, the training method, the device, the electronic device, and the storage medium provided in the embodiments of the present application can obtain the source linear spectrum of a source speaker according to source speech data of the source speaker; input the source linear spectrum into a pre-trained speech coding model to output corresponding spectral feature prediction data, wherein the speech coding model is trained according to a sample phoneme sequence and sample linear spectrum obtained from sample speech data; and input the spectral feature prediction data and target speaker feature data of a target speaker into a speech reconstruction model to output corresponding target speech data, wherein the speech reconstruction model is trained according to sample spectral feature data, sample speaker feature data, and sample speech data, and the sample spectral feature data is obtained according to a sample linear spectrum extracted from the sample speech data. In this way, the content information of the source speech data is decoupled from the speaker feature, and in the training stage and the speech conversion stage of the speech coding model, the input and the output of the speech coding model only contain content information, respectively. The speech reconstruction model is used to reconstruct speech data together with the content information and the speaker feature, which is conducive to improving the training speed and the training effect of the speech coding model, and further improves the speech conversion effect.

[0053] These aspects or other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0055] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present application is shown.

[0056] Figure 2A flowchart of a method for training a speech synthesis model is shown.

[0057] Figure 3 A structural diagram of a speech synthesis device is shown.

[0058] Figure 4 A structural diagram of an electronic device is shown.

[0059] Figure 5 A structural diagram of a storage medium is shown. DETAILED DESCRIPTION

[0060] Embodiments of the present application are described below in detail with reference to the accompanying drawings. The same or similar components are denoted by the same or similar reference numerals throughout the drawings, and repeated description is omitted. The embodiments described below are exemplary and are for the purpose of explanation only, and are not to be understood as limiting the present application.

[0061] In order to make the technical personnel in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0062] In the embodiments of the present application, at least one means one or more, and multiple means two or more. In the description of the present application, the terms "first", "second", "third", etc. are only used for distinguishing the description purpose, and cannot be understood as indicating or implying relative importance, or indicating or implying sequence.

[0063] In the present specification, the reference "one embodiment" or "some embodiments" means that a specific feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Therefore, the terms "include", "contain", "have" and their variants in the present specification mean "include but not limited to", unless otherwise specifically emphasized.

[0064] One embodiment of this application provides a voice conversion method. The execution entity of the voice conversion method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices that can be configured to execute the voice conversion method provided in this embodiment. In other words, the voice conversion method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0065] Please see Figure 1 The diagram shown is a schematic flowchart of a speech conversion method provided in one embodiment of this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 1 The illustrated process sequence is limited. In this embodiment, the speech conversion method includes the following steps:

[0066] S10, Obtain the source linear spectrum of the source speaker based on the source speech data of the source speaker;

[0067] Among these methods, a short-time Fourier transform (STFT) can be performed on the source speech data to extract the source linear spectrum.

[0068] S20, the source linear spectrum is input into a pre-trained speech coding model, and the corresponding spectral feature prediction data is output. The speech coding model is trained based on the sample phoneme sequence and sample linear spectrum obtained from the sample speech data.

[0069] The speech coding model extracts features from the input source linear spectrum, and the output spectral feature prediction data is the mean and variance of a multivariate Gaussian distribution.

[0070] As one implementation method, the training steps of the speech coding model include:

[0071] S21, Obtain at least one training sample, the training sample including a sample phoneme sequence and a sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained based on sample speech data;

[0072] S22, input the sample phoneme sequence to the phoneme encoder and output the phoneme feature data of the sample phoneme sequence;

[0073] The phoneme is the smallest unit in speech, which can be analyzed according to the pronunciation action in the syllable of a word, and one action constitutes one phoneme. For example, in Chinese, there are 32 phonemes, and the phonemes can be divided into initial and final. For example, the source speech data can be subjected to speech recognition to obtain corresponding text content, and the corresponding sample phoneme sequence of the text content can be composed of the initials and finals of each character in turn. Taking the text content of “nǐ hǎo zhōngguó píng'ān” as an example, the corresponding sample phoneme sequence can be “n, i, h, ao, zh, ong, g, uo, p, ing, an”. It should be understood that the corresponding sample phoneme sequence can also be in other forms, and the form of the sample phoneme sequence is not limited in the present application.

[0074] The phoneme encoder maps each character in the sample phoneme sequence to a corresponding character feature vector, and each character feature vector forms the phoneme feature data p[p0, p1, p2, …, p i ,…,p T1-1 ], where 0≤i≤T1-1, T1 is the number of vectors in the phoneme feature data, and T1 is an integer greater than or equal to 2.

[0075] S23, inputting the sample linear spectrum into the speech coding model to output sample spectrum feature prediction data of the sample linear spectrum;

[0076] The speech coding model extracts features from the input sample linear spectrum and outputs sample spectrum feature prediction data of the sample linear spectrum. The sample spectrum feature prediction data q[q0, q1, q2, …, q j ,…,q T2-1 ], where 0≤j≤T2-1, T2 is the number of vectors in the sample spectrum feature prediction data, and T2 is an integer greater than or equal to 2. The sample spectrum feature prediction data is the mean and variance of the multivariate Gaussian distribution.

[0077] S24, obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectrum feature prediction data;

[0078] As an implementation, the phoneme feature data and the sample spectrum feature prediction data are subjected to cross attention to obtain the first alignment matrix. Specifically, the calculation method of the first alignment matrix a is as follows:

[0079]

[0080] wherein a i,j is the matrix element of the i-th row and the j-th column in the first alignment matrix a, p i is the i-th feature vector in the phoneme feature data, and q jLet p be the j-th feature vector in the sample spectral feature prediction data. m Let be the m-th feature vector in the phoneme feature data, D be the dimension of the output of the phoneme encoder and speech coding model, exp() be the exponential function with the natural constant e as the base, and T1 be the length of the phoneme feature data. Based on the first alignment matrix α and the phoneme feature data p, the sample spectral feature prediction data q can be calculated.

[0081] S25, obtain the index mapping vector (IMV) of the sample spectrum feature prediction data based on the first alignment matrix and the phoneme feature data;

[0082] As one implementation method, the index mapping vector π is calculated as follows:

[0083]

[0084] Where, π j Let α be the index mapping vector of the j-th index. i,j p is the matrix element in the i-th row and j-th column of the first alignment matrix α. i Let be the i-th feature vector in the phoneme feature data.

[0085] S26, Obtain spectral feature calculation data based on the index mapping vector;

[0086] As one implementation method, step S26 specifically includes the following steps:

[0087] S261, Obtain the second alignment matrix of the sample spectral feature prediction data based on the index mapping vector and the phoneme feature data;

[0088] As one implementation method, the second alignment matrix γ is calculated as follows:

[0089]

[0090] Where, γ i,j p is the matrix element in the i-th row and j-th column of the second alignment matrix γ. i Let π be the i-th feature vector in the phoneme feature data. j For the j-th index mapping vector, π n Let be the nth index mapping vector, δ be the alignment variance, exp() be an exponential function with the natural constant e as the base, and T2 be the length of the sample spectral feature prediction data. Phoneme feature data p can be calculated based on the second alignment matrix γ and the sample spectral feature prediction data q.

[0091] S262, obtaining alignment position calculation data of the phoneme feature data according to the second alignment matrix and the sample spectrum feature prediction data;

[0092] As an implementation, the calculation of the alignment position calculation data e is as follows:

[0093]

[0094] wherein e i is the i-th alignment position data, γ i,n is the matrix element of the i-th row and the n-th column in the second alignment matrix γ, q n is the n-th feature vector in the sample spectrum feature prediction data.

[0095] S263, obtaining a reconstructed alignment matrix according to the alignment position calculation data and the sample spectrum feature prediction data;

[0096] As an implementation, the reconstructed alignment matrix α' is calculated according to the following formula:

[0097]

[0098] wherein α' i,j is the matrix element of the i-th row and the j-th column in the reconstructed alignment matrix α', q j is the j-th feature vector in the sample spectrum feature prediction data, e i is the i-th alignment position vector, e m is the m-th alignment position vector, δ is an alignment variance, exp() is an exponential function with the natural constant e as the base, T1 is the length of the phoneme feature data, and T1 is also the length of the alignment position calculation data.

[0099] S264, obtaining spectrum feature calculation data according to the reconstructed alignment matrix;

[0100] wherein the reconstructed alignment matrix can be decoded by a decoder, the decoder can be a neural network structure, the reconstructed alignment matrix is input into the decoder, and the decoder outputs the spectrum feature calculation data according to the reconstructed alignment matrix. The spectrum feature calculation data is also the mean and variance of a multivariate Gaussian distribution.

[0101] S27, obtaining a KL (Kullback-Leibler) divergence of the spectrum feature calculation data and the sample spectrum feature prediction data, and adjusting parameters of the speech coding model according to the KL divergence until the speech coding model reaches a training convergence condition;

[0102] In the training process of the speech coding model, the spectral feature calculation data q' and the spectral feature prediction data q are expected to be as close as possible, the spectral feature prediction data q is a predicted value, the spectral feature calculation data q' is a calculated value, the KL divergence of the two can represent the prediction error to adjust the parameters of the speech coding model until the speech coding model reaches the training convergence condition, and the speech coding model reaching the training convergence condition is used as the trained speech coding model for the prediction of the spectral feature data in actual speech conversion.

[0103] As another implementation, the training step of the speech coding model comprises:

[0104] S31, obtaining at least one training sample, the training sample comprising a sample phoneme sequence and a sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained according to sample speech data;

[0105] S32, inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence;

[0106] S33, inputting the sample linear spectrum into the speech coding model to output sample spectral feature prediction data of the sample linear spectrum;

[0107] S34, obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectral feature prediction data;

[0108] The steps S31, S32, S33 and S34 are respectively the steps S21, S22, S23 and S24.

[0109] S35, obtaining a corresponding phoneme index vector according to the phoneme feature data;

[0110] Wherein, the phoneme index vector k = [0, 1, …, T1-1], and the lengths of the phoneme index vector and the phoneme feature data are both T1.

[0111] S36, obtaining the index mapping vector according to the first alignment matrix and the phoneme index vector;

[0112] As an implementation, the index mapping vector π ′ is calculated as follows:

[0113]

[0114] Wherein, π′ j is the jth index mapping vector, a i,j is the matrix element in the i-th row and the j-th column of the first alignment matrix a, k iwhere k is the phoneme index vector, k = [0, 1, …, T1-1], and is the i-th element of the phoneme index vector k, and is the j-th element of the first alignment matrix

[0115] S37, obtaining a reconstruction alignment matrix according to the index mapping vector and the phoneme index vector;

[0116] As an implementation, in order to reduce the problem of error superposition, a bidirectional cumulative operation is designed to generate a second index mapping vector:

[0117] Δπ′ j = π′ j - π′ j-1 , 0 < j < T2-1,

[0118] Δπ j = ReLU(Δπ′ j ), 0 < j < T2-1,

[0119] For the j-th time step, Δπ is cumulatively added in the forward and reverse directions respectively:

[0120]

[0121] Finally, the second index mapping vector is obtained by:

[0122]

[0123] wherein,

[0124]

[0125] The second index mapping vector π* is calculated by the following formula to reconstruct the alignment matrix α′:

[0126]

[0127] wherein, α i,j ′ is the matrix element of the i-th row and the j-th column of the reconstruction alignment matrix α′, k i is the i-th element of the phoneme index vector k, k = [0, 1, …, T1-1], k m is the m-th element of the phoneme index vector k, π j * is the j-th element of the second index mapping vector π*, exp() is the exponential function with the natural constant e as the base, T1 is the length of the phoneme feature data, and σ 2 is a parameter representing the alignment variation coefficient.

[0128] S38, obtaining spectral feature calculation data according to the reconstruction alignment matrix;

[0129] The step S38 specifically refers to the step S26.

[0130] S39, obtaining the KL (Kullback-Leibler) divergence of the spectrum feature calculation data and the sample spectrum feature prediction data, adjusting the parameters of the speech coding model according to the KL divergence until the speech coding model reaches the training convergence condition;

[0131] The step S39 specifically refers to the step S27.

[0132] S30, inputting the spectrum feature prediction data and the target speaker feature data of the target speaker into a speech reconstruction model to output corresponding target speech data, wherein the speech reconstruction model is obtained by training sample spectrum feature data, sample speaker feature data and sample speech data, and the sample spectrum feature data is obtained from the sample linear spectrum extracted from the sample speech data.

[0133] Wherein, the speaker feature can include the timbre feature, which is obtained by feature extraction from the speech data of the speaker, and the speaker feature can be extracted by using one-hot encoding, d-vector or x-vector encoder for feature extraction of the speech data.

[0134] As an embodiment, the step S30 specifically includes the following steps:

[0135] S41, splicing the spectrum feature prediction data and the target speaker feature data of the target speaker to obtain spliced feature data;

[0136] Wherein, the splicing of the two feature data can be realized by summing the spectrum feature prediction data and the target speaker feature data of the target speaker.

[0137] S42, obtaining the corresponding target speech data according to the spliced feature data;

[0138] Wherein, the target speech data is generated by waveform reconstruction according to the spliced feature data, and the target speech data carries the timbre information of the target speaker.

[0139] As an embodiment, the training step of the speech reconstruction model includes:

[0140] S51, obtaining at least one training sample, the training sample including sample speech data, sample linear spectrum extracted from the sample speech data and sample speaker feature data;

[0141] S52, inputting the sample linear spectrum into a speech coding model to output sample spectrum feature prediction data of the sample linear spectrum;

[0142] S53, inputting the sample spectrum feature prediction data and the sample speaker feature data into a speech reconstruction model, and outputting reconstructed sample speech data;

[0143] S54, calculating a reconstruction loss according to the sample speech data and the reconstructed sample speech data, adjusting parameters of the speech reconstruction model according to the reconstruction loss, and until the speech reconstruction model reaches a training convergence condition.

[0144] Please refer to Figure 2 the flowchart of the training method of the speech conversion model provided by an embodiment of the present application. It should be noted that the method of the present application is not limited to the order of the flowchart as long as the same result is achieved. In the embodiment, the speech conversion model includes a speech coding model and a speech reconstruction model, and the training method of the speech conversion model includes the following steps: Figure 2

[0145] S61, obtaining at least one training sample, the training sample including sample speech data and sample linear spectrum, sample phoneme sequence and sample speaker feature data extracted from the sample speech data;

[0146] S62, inputting the sample phoneme sequence into a phoneme encoder, and outputting phoneme feature data of the sample phoneme sequence;

[0147] S63, inputting the sample linear spectrum into the speech coding model, and outputting sample spectrum feature prediction data of the sample linear spectrum;

[0148] S64, obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectrum feature prediction data;

[0149] S65, obtaining an index mapping vector of the sample spectrum feature prediction data according to the first alignment matrix and the phoneme feature data;

[0150] S66, obtaining spectrum feature calculation data according to the index mapping vector;

[0151] S67, obtaining the KL divergence of the spectrum feature calculation data and the sample spectrum feature prediction data, adjusting parameters of the speech coding model according to the KL divergence, and until the speech coding model reaches a training convergence condition;

[0152] S68, inputting the sample spectrum feature prediction data and the sample speaker feature data into a speech reconstruction model, and outputting reconstructed sample speech data;

[0153] ​S69, calculate a reconstruction loss according to the sample voice data and the reconstructed sample voice data, and adjust parameters of the voice reconstruction model according to the reconstruction loss until the voice reconstruction model reaches a training convergence condition.

[0154] The steps of the training method of the embodiment can be specifically referred to the description of the above embodiments.

[0155] As a first implementation, step S65 refers to step S25, and step S66 refers to step S26.

[0156] As a second implementation, step S65 refers to steps S35 and S36, and step S66 refers to steps S37 and S38.

[0157] As shown in Figure 3 An embodiment of the present application provides a voice conversion device, which comprises an extraction module 31, an encoding module 32 and a reconstruction module 33. The extraction module 31 is configured to obtain source linear spectrum of a source speaker according to source voice data of the source speaker. The encoding module 32 is configured to input the source linear spectrum into a pre-trained voice encoding model, and output corresponding spectral feature prediction data. The voice encoding model is obtained by training sample phoneme sequences and sample linear spectrum obtained from sample voice data. The reconstruction module 33 is configured to input the spectral feature prediction data and target speaker feature data of a target speaker into a voice reconstruction model, and output corresponding target voice data. The voice reconstruction model is obtained by training sample spectral feature data, sample speaker feature data and sample voice data. The sample spectral feature data is obtained according to sample linear spectrum extracted from the sample voice data.

[0158] As an implementation form, the encoding module 32 is further configured to: obtain at least one training sample, the training sample comprising a sample phoneme sequence and a sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained according to sample speech data; input the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence; input the sample linear spectrum into the speech coding model to output sample spectral feature prediction data of the sample linear spectrum; obtain a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectral feature prediction data; obtain an index mapping vector of the sample spectral feature prediction data according to the first alignment matrix and the phoneme feature data; obtain spectral feature calculation data according to the index mapping vector; obtain a KL divergence of the spectral feature calculation data and the sample spectral feature prediction data, and adjust parameters of the speech coding model according to the KL divergence until the speech coding model reaches a training convergence condition.

[0159] As an implementation form, the encoding module 32 is further configured to: obtain a second alignment matrix of the sample spectral feature prediction data according to the index mapping vector and the phoneme feature data; obtain alignment position calculation data of the phoneme feature data according to the second alignment matrix and the sample spectral feature prediction data; obtain a reconstruction alignment matrix according to the alignment position calculation data and the sample spectral feature prediction data; and obtain spectral feature calculation data according to the reconstruction alignment matrix and the phoneme feature data.

[0160] As an implementation form, the encoding module 32 is further configured to: obtain a corresponding phoneme index vector according to the phoneme feature data; obtain the index mapping vector according to the first alignment matrix and the phoneme index vector; correspondingly, the encoding module 32 is further configured to: obtain a reconstruction alignment matrix according to the index mapping vector and the phoneme index vector; and obtain spectral feature calculation data according to the reconstruction alignment matrix and the phoneme feature data.

[0161] As an implementation form, the reconstruction module 33 is further configured to: obtain at least one training sample, the training sample comprising sample speech data, sample linear spectrum extracted from the sample speech data, and sample speaker feature data; input the sample linear spectrum into a speech coding model to output sample spectral feature prediction data of the sample linear spectrum; input the sample spectral feature prediction data and the sample speaker feature data into a speech reconstruction model to output reconstructed sample speech data; calculate a reconstruction loss according to the sample speech data and the reconstructed sample speech data, and adjust parameters of the speech reconstruction model according to the reconstruction loss until the speech reconstruction model reaches a training convergence condition.

[0162] As an implementation form, the reconstruction module 33 is further configured to splice the spectral feature prediction data and target speaker feature data of the target speaker to obtain spliced feature data, and acquire the target speech data corresponding to the spliced feature data.

[0163] Figure 4 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. As shown in the figure, the electronic device 60 includes a processor 61 and a memory 62 coupled to the processor 61. Figure 4 The memory 62 stores program instructions for implementing the voice conversion method or the training method of the voice conversion model of any of the above embodiments.

[0164] The memory 62 stores program instructions for implementing the voice conversion method or the training method of the voice conversion model of any of the above embodiments.

[0165] The processor 61 is configured to execute the program instructions stored in the memory 62 to perform voice conversion or training of the voice conversion model.

[0166] The processor 61 can also be referred to as a CPU (Central Processing Unit). The processor 61 can be an integrated circuit chip having a processing capability of signals. The processor 61 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0167] Referring to Figure 5 , Figure 5 FIG. 2 is a structural schematic diagram of a storage medium according to an embodiment of the present application. The storage medium 70 according to the embodiment of the present application stores program instructions 71 capable of implementing all the above methods. The program instructions 71 can be stored in the above storage medium in the form of a software product, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media capable of storing program codes, or a terminal device such as a computer, a server, a mobile phone, a tablet, etc.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiment is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0169] In addition, each function unit in the embodiments of the present application can be integrated into a processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware, or can be implemented in the form of a software function unit. The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

[0170] The above is only a preferred embodiment of the present application, and does not limit the present application in any form, although the present application has disclosed the above as a preferred embodiment, however, it is not intended to limit the present application, any person skilled in the art, without departing from the scope of the technical solution of the present application, can make some changes or modifications as equivalent embodiments with the above disclosed technical content, but as long as it does not deviate from the technical solution content of the present application, any brief introduction, modification, equivalent change and modification of the above embodiments according to the technical essence of the present application, are still within the scope of the technical solution of the present application.

Claims

1. A voice conversion method characterized by, The method comprises the following steps: obtaining source linear spectrum of source speaker according to source speech data of source speaker; inputting the source linear spectrum into a pre-trained speech coding model to output corresponding spectral feature prediction data, wherein the speech coding model is trained according to sample phoneme sequence and sample linear spectrum obtained from sample speech data; the training steps of the speech coding model comprise: obtaining at least one training sample, the training sample comprising sample phoneme sequence and sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained from sample speech data; inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence, wherein the phoneme encoder maps each character in the sample phoneme sequence into a corresponding character feature vector, and each character feature vector forms the phoneme feature data; inputting the sample linear spectrum into the speech coding model to output sample spectral feature prediction data of the sample linear spectrum; obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectral feature prediction data; obtaining an index mapping vector of the sample spectral feature prediction data according to the first alignment matrix and the phoneme feature data; obtaining spectral feature calculation data according to the index mapping vector; obtaining the KL divergence of the spectral feature calculation data and the sample spectral feature prediction data, and adjusting the parameters of the speech coding model according to the KL divergence until the speech coding model reaches a training convergence condition; inputting the spectral feature prediction data and target speaker feature data of target speaker into a speech reconstruction model to output corresponding target speech data, wherein the speech reconstruction model is trained according to sample spectral feature data, sample speaker feature data and sample speech data, and the sample spectral feature data is obtained from the sample linear spectrum extracted from the sample speech data.

2. The voice conversion method of claim 1, wherein, The method comprises the following steps: obtaining a second alignment matrix of the sample spectral feature prediction data according to the index mapping vector and the phoneme feature data; obtaining alignment position calculation data of the phoneme feature data according to the second alignment matrix and the sample spectral feature prediction data; obtaining a reconstruction alignment matrix according to the alignment position calculation data and the sample spectral feature prediction data; obtaining spectral feature calculation data according to the reconstruction alignment matrix.

3. The voice conversion method of claim 1, wherein, The method comprises the following steps: obtaining a corresponding phoneme index vector according to the phoneme feature data; obtaining the index mapping vector according to the first alignment matrix and the phoneme index vector; Accordingly, the method comprises the following steps: obtaining a reconstruction alignment matrix according to the index mapping vector and the phoneme index vector; obtaining spectral feature calculation data according to the reconstruction alignment matrix.

4. The voice conversion method of claim 1, wherein, The training steps of the speech reconstruction model comprise: obtaining at least one training sample, the training sample comprising sample speech data and sample linear spectrum and sample speaker feature data extracted from the sample speech data; inputting the sample linear spectrum into a speech coding model to output sample spectral feature prediction data of the sample linear spectrum; inputting the sample spectral feature prediction data and the sample speaker feature data into a speech reconstruction model to output reconstructed sample speech data; calculating a reconstruction loss according to the sample speech data and the reconstructed sample speech data, and adjusting parameters of the speech reconstruction model according to the reconstruction loss until the speech reconstruction model reaches a training convergence condition.

5. The voice conversion method of claim 1, wherein, The inputting the spectral feature prediction data and the target speaker feature data of the target speaker into the speech reconstruction model to output corresponding target speech data comprises: splicing the spectral feature prediction data and the target speaker feature data of the target speaker to obtain spliced feature data; obtaining the corresponding target speech data according to the spliced feature data. 6.A method for training a speech conversion model, the method comprising: The speech conversion model comprises a speech coding model and a speech reconstruction model, and the training method comprises: obtaining at least one training sample, the training sample comprising sample speech data and sample linear spectrum, sample phoneme sequence and sample speaker feature data extracted from the sample speech data; inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence, wherein the phoneme encoder maps each character in the sample phoneme sequence to a corresponding character feature vector, and each character feature vector forms the phoneme feature data; inputting the sample linear spectrum into the speech coding model to output sample spectral feature prediction data of the sample linear spectrum; obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectral feature prediction data; obtaining an index mapping vector of the sample spectral feature prediction data according to the first alignment matrix and the phoneme feature data; obtaining spectral feature calculation data according to the index mapping vector; obtaining the KL divergence of the spectral feature calculation data and the sample spectral feature prediction data, adjusting parameters of the speech coding model according to the KL divergence until the speech coding model reaches a training convergence condition; inputting the sample spectral feature prediction data and the sample speaker feature data into a speech reconstruction model to output reconstructed sample speech data; calculating a reconstruction loss according to the sample speech data and the reconstructed sample speech data, and adjusting parameters of the speech reconstruction model according to the reconstruction loss until the speech reconstruction model reaches a training convergence condition.

7. A voice conversion apparatus characterized by comprising: comprises: an extraction module configured to obtain a source linear spectrum of a source speaker according to source speech data of the source speaker; The encoding module is configured to input the source linear spectrum into a pre-trained speech encoding model to output corresponding spectral feature prediction data, wherein the speech encoding model is trained according to a sample phoneme sequence and sample linear spectrum obtained from sample speech data; the training step of the speech encoding model comprises: obtaining at least one training sample, the training sample comprising a sample phoneme sequence and a sample linear spectrum corresponding to the sample phoneme sequence, the sample phoneme sequence and the sample linear spectrum being obtained according to sample speech data; inputting the sample phoneme sequence into a phoneme encoder to output phoneme feature data of the sample phoneme sequence, wherein the phoneme encoder maps each character in the sample phoneme sequence to a corresponding character feature vector, and each character feature vector forms the phoneme feature data; inputting the sample linear spectrum into the speech encoding model to output sample spectral feature prediction data of the sample linear spectrum; obtaining a first alignment matrix of the phoneme feature data according to the phoneme feature data and the sample spectral feature prediction data; obtaining an index mapping vector of the sample spectral feature prediction data according to the first alignment matrix and the phoneme feature data; obtaining spectral feature calculation data according to the index mapping vector; obtaining the KL divergence of the spectral feature calculation data and the sample spectral feature prediction data, and adjusting the parameters of the speech encoding model according to the KL divergence until the speech encoding model reaches a training convergence condition; The reconstruction module is configured to input the spectral feature prediction data and target speaker feature data of a target speaker into a speech reconstruction model to output corresponding target speech data, wherein the speech reconstruction model is trained according to sample spectral feature data, sample speaker feature data and sample speech data, and the sample spectral feature data is obtained according to a sample linear spectrum extracted from the sample speech data.

8. An electronic device, comprising: The processor, and a memory coupled to the processor, the memory storing program instructions executable by the processor; the processor executes the program instructions stored in the memory to implement the speech conversion method according to any one of claims 1 to 5 or implement the training method of the speech conversion model according to claim 7.

9. A storage medium, characterized by The storage medium stores program instructions, and the program instructions are executed by the processor to implement the speech conversion method according to any one of claims 1 to 5 or implement the training method of the speech conversion model according to claim 7.

Images