Speech conversion model training method and device, speech conversion method and device

Abstract

The present invention relates to a speech conversion model training method and device in the field of speech conversion, a speech conversion method and device, and a model training method, comprising the steps of: obtaining a first speech and text data with the same content as the first speech, calculating a first content feature with the text data; extracting a spectral feature of the first speech, outputting a first spectral feature, calculating a first speaker feature and a first latent variable through the first spectral feature; inputting the first latent variable and the first speaker feature into a stream model, calculating and outputting a second speech feature with the first speaker feature as a condition, calculating a loss function with the second speech feature and the first content feature, extracting a first latent variable after reaching a preset optimization parameter, and inputting the optimized first latent variable into a decoder to obtain a predicted speech. The technology of the present invention well retains information such as the speaker's tone and intonation.

Description

Technical Field

[0001] The present invention relates to the field of speech conversion, and in particular to a speech conversion model training method and device, and a speech conversion method and device. Background Art

[0002] Due to the development of deep learning and its application in various fields, speech conversion has also benefited a lot. Speech conversion is to convert the timbre of speech. Its goal is to change only the speaker's timbre, and the speaker's content, emotion, tone, speed, etc. should be consistent with the original audio. For example: there are two speakers A and B. At this time, A said a sentence (S). The function of speech conversion is to change the timbre of this sentence (S) to B's voice, and the other content remains unchanged. According to the data set used for training, it can be divided into: 1. Speech conversion based on parallel corpus, 2. Speech conversion based on non-parallel corpus. Parallel corpus means: for each sentence (S1) in the data set, there is another sentence (S2). The only difference between them is that the speaker's timbre is inconsistent, and other information, such as content, emotion, tone, speed, etc. are the same. Since this kind of parallel corpus is difficult to obtain, the current research focus is on speech conversion based on non-parallel corpus.

[0003] Content coding, existing speech conversion technology first extracts comparative prediction coding from the source audio using speech recognition. Comparative prediction coding generally does not contain information such as the speaker, tone, and intonation in the audio, but contains more content information.

[0004] Speaker encoding, Speaker encoding refers to the technology of extracting speakers from audio, generally using deep learning technology to extract a speaker vector.

[0005] Feature decoding,Feature decoding is to fuse the content encoding and speaker encoding through a deep learning network, and calculate the loss with the Mel spectrum extracted from the real speech.

[0006] The vocoder uses the Mel spectrum extracted from the real speech as input and uses neural network models such as WaveNet, Parallel WaveNet, Hifi-Gan, etc. to predict the real speech waveform. The input in the inference stage is the Mel spectrum after the source audio is converted, not the real Mel spectrum.

[0007] The existing technical route is: 1. Identify content encoding through the speech recognition framework. 2. Extract speaker vector from the pre-trained model. In the training stage, 1 and 2 are decoded to obtain the Mel spectrum of the source audio. In the inference stage, the speaker vector in 2 is replaced with the speaker vector of the target speaker. The disadvantage is that the recognition of content depends on the speech recognition model, and the converted audio can only retain the content information, and the tone and intonation cannot be converted. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a framework for encoding content without relying on speech recognition and retaining information such as tone and intonation.

[0009] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0010] A speech conversion model training method comprises the steps of: obtaining a first speech and text data having the same content as the first speech, and calculating a first content feature with the text data;

[0011] Extracting a spectral feature of a first speech, outputting a first spectral feature, and calculating a first speaker feature and a first latent variable through the first spectral feature;

[0012] The first latent variable and the first speaker feature are input into the stream model, and the second speech feature is calculated and output based on the first speaker feature. The loss function is calculated based on the second speech feature and the first content feature, and the first latent variable after reaching the preset optimization parameter is extracted. The optimized first latent variable is input into the decoder to obtain the predicted speech.

[0013] Preferably, the specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature and the first latent variable through the first spectral feature includes:

[0014] A posteriori encoder is used to calculate a first latent variable using the first spectrum feature, and the posteriori encoder includes a plurality of WaveNet residual models.

[0015] Preferably, the specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature and the first latent variable through the first spectral feature includes:

[0016] A speaker encoder is used to calculate a first speaker feature using the first spectrum feature, and the speaker encoder includes a Conformer model.

[0017] Preferably, the flow model includes several WaveNet residual blocks for constructing a mapping relationship between content features and latent variables.

[0018] The content features are converted into latent variables through the flow model, and the latent variables are converted into content features through the flow model.

[0019] Preferably, the method for calculating the first content feature using the text data includes:

[0020] The text data is obtained by using the glyphs to obtain the phonemes corresponding to the text, the phonemes of the text are characterized, and the CBHG module encodes the characterized features to obtain the first content features.

[0021] The present invention also provides a speech conversion method, comprising a stream model trained according to the speech conversion model training method, and further comprising the steps of:

[0022] Acquire a first audio feature P1 that is unrelated to source audio speaker information;

[0023] Acquire the speech of the target speaker to be converted, extract the frequency spectrum feature of the speech of the target speaker, output the second frequency spectrum feature, and calculate the second speech feature S2 according to the second frequency spectrum feature;

[0024] The second speech feature and the first audio feature are input into a stream model to obtain a second latent variable Z2, and the second latent variable is decoded to generate a target audio.

[0025] The method for obtaining a first audio feature that is independent of source audio speaker information includes performing code conversion using a content encoder.

[0026] Preferably, the speaker information includes the speaker's timbre.

[0027] Preferably, the method for obtaining the first audio feature that is unrelated to the source audio speaker information includes:

[0028] The first latent variable and the second speaker feature are input into the stream model, and the second speaker feature P1 is calculated and output based on the second speaker feature.

[0029] The present invention also provides a speech conversion model training device, comprising: a main controller, which obtains a first speech, calculates a first spectrum feature, calculates text data identical to the first speech, and controls the input and output of data between a content encoder, a posterior encoder, a stream model unit, and a decoder;

[0030] A content encoder is configured to obtain text data identical to the first voice content and calculate a first content feature using the text data;

[0031] A posterior encoder receives a first spectrum feature and calculates a first latent variable using the first spectrum feature;

[0032] A speaker encoder receives a first frequency spectrum feature and calculates a first speaker feature according to the first frequency spectrum feature;

[0033] A stream model unit receives a first latent variable and the first speaker feature, calculates and outputs a second speech feature based on the first speaker feature, calculates a loss function based on the second speech feature and the first content feature, and extracts the first latent variable after reaching a preset optimization parameter;

[0034] The decoder is used to input the optimized first latent variable into the decoder to obtain the predicted speech.

[0035] The present invention also provides a voice conversion device, comprising:

[0036] A speech conversion model training device, comprising:

[0037] A content encoder encodes the text content of the source speaker's speech through a deep learning model to obtain a first audio feature that is independent of the source speaker's information;

[0038] A speaker encoder receives a target speaker speech to be converted, extracts a spectral feature of the target speaker speech, outputs a second spectral feature, and calculates a second speech feature through the second spectral feature;

[0039] A stream model unit receives a second speech feature and the first audio feature, and outputs a second latent variable using the second speech feature as a condition;

[0040] The decoder receives the second latent variable and outputs the target audio.

[0041] Beneficial effects of the present invention:

[0042] The present invention avoids the shortcomings of the prior art, does not need to rely on the framework of speech recognition to encode the content, and can also retain information such as tone and intonation. Since the structure of the present invention can well decouple the speaker and non-speaker features in the spectrum, the converted audio retains other information except the speaker's timbre.

[0043] In addition, the present invention is a universal speech conversion technology that can convert source audio of any language well. By extracting spectral features, the correspondence at the frame level is more precise than the correspondence at the phoneme level, so even if the training set only contains Chinese data sets, it can also convert English, Japanese, Korean, Spanish, dialects, etc. well. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 is a flow chart of the speech conversion model training method of embodiment 1;

[0046] Figure 2 It is a flow chart of the voice conversion method of embodiment 2. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0048] Embodiment 1:

[0049] A speech conversion model training method,

[0050] Acquire a first voice and text data having the same content as the first voice, and calculate a first content feature C using the text data;

[0051] Extracting the spectral features of the first speech, outputting the first spectral features, and calculating the first speaker feature S1 and the first latent variable Z1 through the first spectral features;

[0052] The first latent variable and the first speaker feature are input into the stream model, and the second speech feature P1 is calculated and output based on the first speaker feature. The loss function is calculated based on the second speech feature and the first content feature, and the first latent variable after reaching the preset optimization parameter is extracted. The optimized first latent variable is input into the decoder to obtain the predicted speech.

[0053] The specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature S1 and the first latent variable Z1 through the first spectral feature includes:

[0054] A posteriori encoder is used to calculate a first latent variable Z1 using the first spectrum feature, and the posteriori encoder includes several WaveNet residual models.

[0055] The specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature S1 and the first latent variable Z1 through the first spectral feature includes:

[0056] A speaker encoder is used to calculate a first speaker feature using the first spectrum feature, and the speaker encoder includes a Conformer model.

[0057] The flow model includes several residual blocks of WaveNet, which are used to construct a mapping relationship between content features and latent variables. The content features are converted into latent variables through the flow model, and the latent variables are converted into content features through the flow model.

[0058] As a preferred solution, the method for calculating the first content feature C using the text data includes:

[0059] The text data is obtained by using the glyphs to obtain the phonemes corresponding to the text, the phonemes of the text are characterized, and the CBHG module encodes the characterized features to obtain the first content features.

[0060] Step 1: Use the grapheme-to-phoneme tool to get the pinyin of the text, and then split the initial consonants and finals of the pinyin to get the phonemes corresponding to the text.

[0061] Step 2: All phonemes are organized into a phoneme dictionary. The number of phoneme dictionaries is used as the dimension of the embedding layer to represent the phonemes of the text.

[0062] Step 3: The represented features are encoded by the CBHG module, which includes a recurrent neural network of a one-dimensional convolutional filter bank, a highway network, and a bidirectional gated recurrent unit.

[0063] As a preferred solution, the structure of the decoder is the same as the generator structure of HiFi-GAN.

[0064] Embodiment 2:

[0065] A speech conversion method includes a stream model trained according to the speech conversion model training method disclosed in Example 1, wherein a first speech is a source speech and a second speech is a target person's speech, that is, the first speaker information of the first speech is removed and the first speech is converted into a target speech with the target person's speech information of the second speech.

[0066] The following steps are involved:

[0067] Acquire a first audio feature P1 that is unrelated to source audio speaker information;

[0068] Acquire the speech of the target speaker to be converted, extract the frequency spectrum feature of the speech of the target speaker, output the second frequency spectrum feature, and calculate the second speech feature S2 according to the second frequency spectrum feature;

[0069] The second speech feature and the first audio feature are input into a stream model to obtain a second latent variable Z2, and the second latent variable is decoded to generate a target audio.

[0070] The method for obtaining a first audio feature that is independent of source audio speaker information includes performing code conversion using a content encoder.

[0071] Example 3

[0072] A speech conversion model training device, comprising:

[0073] A main controller, which obtains a first speech, calculates a first spectrum feature, calculates text data identical to the first speech, and controls input and output of data between a content encoder, a posterior encoder, a stream model unit, and a decoder;

[0074] A content encoder is configured to obtain text data identical to the first voice content and calculate a first content feature using the text data;

[0075] The text content of the source speaker's speech is encoded through a deep learning model to obtain content encoding information that is independent of the source speaker's information;

[0076] A posterior encoder receives a first spectrum feature and calculates a first latent variable using the first spectrum feature;

[0077] A speaker encoder receives a first frequency spectrum feature and calculates a first speaker feature according to the first frequency spectrum feature;

[0078] A stream model unit receives a first latent variable and the first speaker feature, calculates and outputs a second speech feature based on the first speaker feature, calculates a loss function based on the second speech feature and the first content feature, and extracts the first latent variable after reaching a preset optimization parameter;

[0079] The decoder is used to input the optimized first latent variable into the decoder to obtain the predicted speech.

[0080] The trained stream model unit realizes the conversion between content features and latent variables, that is, the content features can be input to the stream model unit, and the stream model unit outputs latent variables.

[0081] Example 4

[0082] A speech conversion model training device, comprising:

[0083] Content encoder, which encodes the text content of the source speaker's speech through a deep learning model to obtain content encoding information that is independent of the source speaker's information;

[0084] A target encoder extracts a target speaker feature vector S2 from the target speaker's speech;

[0085] A stream model unit, wherein the stream model unit in this embodiment is a trained stream model unit, which realizes the conversion between content features and latent variables;

[0086] The decoder takes in latent variable parameters and outputs the target speaker audio.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0088] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A speech conversion model training method, characterized in that: Acquire a first voice and text data having the same content as the first voice, and calculate a first content feature using the text data; Extracting a spectral feature of a first speech, outputting a first spectral feature, and calculating a first speaker feature and a first latent variable through the first spectral feature; Input the first latent variable and the first speaker feature into the stream model, calculate and output the second speech feature based on the first speaker feature, calculate the loss function based on the second speech feature and the first content feature, extract the first latent variable after reaching the preset optimization parameter, and input the optimized first latent variable into the decoder to obtain the predicted speech; The flow model includes several residual blocks of WaveNet, which are used to construct a mapping relationship between content features and latent variables. The content features are converted into latent variables through the flow model, and the latent variables are converted into content features through the flow model.

2. The speech conversion model training method according to claim 1, characterized in that: The specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature and the first latent variable through the first spectral feature includes: A posteriori encoder is used to calculate a first latent variable using the first spectrum feature, and the posteriori encoder includes a plurality of WaveNet residual models.

3. The speech conversion model training method according to claim 1, characterized in that: The specific method of extracting the spectral feature of the first speech, outputting the first spectral feature, and calculating the first speaker feature and the first latent variable through the first spectral feature includes: A speaker encoder is used to calculate a first speaker feature using the first spectrum feature, and the speaker encoder includes a Conformer model.

4. The speech conversion model training method according to claim 1, characterized in that: The method for calculating the first content feature using the text data includes: The text data is obtained by using the glyphs to obtain the phonemes corresponding to the text, the phonemes of the text are characterized, and the CBHG module encodes the characterized features to obtain the first content features.

5. A speech conversion method, characterized in that: The method comprises the following steps: Acquire a first audio feature P1 that is unrelated to source audio speaker information; Acquire the speech of the target speaker to be converted, extract the frequency spectrum feature of the speech of the target speaker, output the second frequency spectrum feature, and calculate the second speech feature S2 according to the second frequency spectrum feature; Inputting the second speech feature and the first audio feature into a stream model to obtain a second latent variable Z2, and decoding the second latent variable Z2 to generate a target audio; The method for obtaining a first audio feature that is independent of source audio speaker information includes performing code conversion using a content encoder.

6. The method for voice conversion according to claim 5, characterized in that: The speaker information includes the speaker's timbre.

7. The method for voice conversion according to claim 5, characterized in that: The method for obtaining a first audio feature that is independent of source audio speaker information includes: The first latent variable and the second speaker feature are input into the stream model, and the second speaker feature P1 is calculated and output based on the second speaker feature.

8. A speech conversion model training device, characterized in that: include: A main controller, which obtains a first speech, calculates a first spectrum feature, calculates text data identical to the first speech, and controls input and output of data between a content encoder, a posterior encoder, a stream model unit, and a decoder; A content encoder is configured to obtain text data identical to the first voice content and calculate a first content feature using the text data; A posterior encoder receives a first spectrum feature and calculates a first latent variable using the first spectrum feature; A speaker encoder receives a first frequency spectrum feature and calculates a first speaker feature according to the first frequency spectrum feature; A stream model unit receives a first latent variable and the first speaker feature, calculates and outputs a second speech feature based on the first speaker feature, calculates a loss function based on the second speech feature and the first content feature, and extracts the first latent variable after reaching a preset optimization parameter; The decoder is used to input the optimized first latent variable into the decoder to obtain the predicted speech.

Images