Speech conversion methods, devices, computer equipment, and storage media

Abstract

This application relates to a speech conversion method, apparatus, computer device, and storage medium. The method includes: acquiring source speech language coding features and source speech fundamental frequency features, and concatenating the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features; acquiring target speech timbre coding features; inputting the source speech concatenation features and the target speech timbre coding features into a decoder for decoding to obtain converted timbre Mel spectrum features; and inputting the converted timbre Mel spectrum features into a vocoder to synthesize converted speech. This application introduces multi-head attention to align non-speaker-related language features between text and audio, and embeds target speaker language features through a decoder by combining source speaker-related features and fundamental frequency features, achieving fast and efficient end-to-end non-autoregressive speech conversion.

Description

Technical Field

[0001] This application relates to the field of voice conversion technology, and particularly to a voice conversion method, apparatus, computer device, and storage medium. Background Art

[0002] Voice Conversion is to transfer the timbre of the original voice speaker to the target voice speaker on the basis of the unchanged speech content, and has important applications in aspects such as movie dubbing, character imitation, and replicating the timbre of characters. It also has rich application scenarios in the vehicle field, such as navigation voice change, IP customization, personalized user timbre customization, etc. By copying the timbre through single-sample voice conversion, not only can the requirements for training data be greatly reduced, but also computing resources can be significantly saved.

[0003] Currently, great progress has been made in voice conversion to a specific target speaker based on deep learning. For example, voice conversion methods based on Cycle GAN, VAE, and ASR can all well achieve voice conversion to speakers within the training set. However, if you want to add the timbre of a target speaker or perform custom replication of the user's timbre, usually a large amount of speaker data is required to retrain a voice conversion model with the timbre of this speaker as the target timbre, or adaptively train the existing model with a small amount of data. In practical applications, the cycle and cost of database recording are relatively high, and it is also difficult for ordinary users to obtain a large amount of voice data of users.

[0004] Although the attention model based on text assistance can share attention with the TTS model to achieve one-to-many conversion, it also has disadvantages such as a long training cycle, large computational complexity, and inability to achieve ultra-long text synthesis conversion. Although the streaming text-to-speech (tts) model can synthesize speech in parallel, it also has the disadvantage of a large number of parameters. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a voice conversion method, apparatus, computer device, and storage medium that can solve the technical problems of large number of parameters, high cycle, and high cost in converting streaming text to the voice of a certain target speaker.

[0006] On the one hand, a voice conversion method is provided, and the method includes:

[0007] Obtain the source speech language encoding feature and the source speech fundamental frequency feature, and splice the source speech language encoding feature and the source speech fundamental frequency feature to form a source speech spliced feature;

[0008] Obtain the target speech timbre encoding feature;

[0009] Input the source speech splicing feature and the target speech timbre encoding feature into the decoder for decoding to obtain the converted timbre Mel spectrogram feature;

[0010] Input the converted timbre Mel spectrogram feature into a vocoder to synthesize the converted speech.

[0011] In one embodiment, the step of inputting the source speech splicing feature and the target speech timbre encoding feature into the decoder for decoding to obtain the converted timbre Mel spectrogram feature includes:

[0012] Input the source speech splicing feature into a one-dimensional convolutional layer for convolutional processing and then input it into a generation module;

[0013] Perform conditional batch normalization on the target speech timbre encoding feature and then input it into the generation module;

[0014] Input the source speech splicing feature after convolutional processing and the output result of the target speech timbre encoding feature after conditional batch normalization processed by the piecewise linear function, upsampling layer, one-dimensional convolutional layer and dilated convolutional layer in the generation module into a one-dimensional convolutional layer for convolutional processing and then output the converted timbre Mel spectrogram feature.

[0015] In one embodiment, the step of inputting the source speech splicing feature and the target speech timbre encoding feature into the decoder for decoding to obtain the converted timbre Mel spectrogram feature further includes:

[0016] Input the noise vector after being processed by a linear layer into a batch normalization layer;

[0017] Input the source speech splicing feature after convolutional processing and the target speech timbre encoding feature after conditional batch normalization into the batch normalization layer, and splice and accumulate them with the noise vector;

[0018] Input the spliced feature after being spliced and accumulated with the noise vector into a one-dimensional convolutional layer for convolutional processing after being processed by the piecewise linear function, upsampling layer, one-dimensional convolutional layer and dilated convolutional layer in the generation module and then output the converted timbre Mel spectrogram feature.

[0019] In one embodiment, the step of inputting the source speech splicing feature after convolutional processing and the target speech timbre encoding feature after conditional batch normalization into the batch normalization layer and splicing and accumulating them with the noise vector includes:

[0020] Take the source speech splicing feature after convolutional processing and the target speech timbre encoding feature after conditional batch normalization as the first speech input feature;

[0021] The generation module is configured with multiple batch normalization layers, multiple piecewise linear functions and upsampling layers. The output of one normalization layer is connected to the input of the piecewise linear function and upsampling layer. The output of the piecewise linear function and upsampling layer is connected to the input of a one-dimensional convolutional layer or a dilated convolutional layer. The output of the one-dimensional convolutional layer or the dilated convolutional layer is connected to the input of another normalization layer.

[0022] The first speech input feature is input into the batch normalization layer and concatenated with the noise vector and accumulated. After processing by the piecewise linear function, the upsampling layer, and the one-dimensional convolutional layer, the first concatenated feature is output.

[0023] The first speech input feature is processed by a piecewise linear function, an upsampling layer, and a one-dimensional convolutional layer to output the first corrected spliced ​​feature.

[0024] The first corrected splicing feature is used to correct the first splicing feature to form the second speech input feature;

[0025] The second speech input feature is input into the batch normalization layer and concatenated with the noise vector. After processing by the piecewise linear function, upsampling layer, and dilated convolutional layer, the second concatenated feature is output.

[0026] The second speech input feature is processed by a piecewise linear function, an upsampling layer, and a dilated convolutional layer to output the second corrected splicing feature.

[0027] The second correction splicing feature is used to correct the second splicing feature to form the speech output feature.

[0028] In one embodiment, the step of obtaining source speech language coding features and source speech fundamental frequency features, and concatenating the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features includes:

[0029] Convert a text sequence into a phoneme sequence;

[0030] The phoneme sequence is input into the encoder pre-trained network, concatenated with the position code, and then input into the encoder multi-head attention layer. After being processed by the residual and normalization layer, the feedforward layer, the residual and normalization layer, and the linear layer, the text sequence features are output.

[0031] The source speech Mel spectrum is input into the decoder pre-trained network, concatenated with the positional coding features to form a concatenated feature input to the masking multi-head attention layer, and after residual and normalization processing, the decoded spectral features are formed and input into the decoder multi-head attention layer.

[0032] The text sequence features are concatenated with the target speech timbre encoding features to form a two-dimensional feature matrix sequence, which is then input into the multi-head attention layer of the decoder.

[0033] The decoder's multi-head attention layer performs alignment operations on the two-dimensional feature matrix sequence and the decoded spectral features, and then outputs the speech of the text to be synthesized after residual and normalization processing.

[0034] The text sequence features are multiplied by the speech of the text to be synthesized to output the source speech splicing features.

[0035] In one embodiment, the step of inputting the text sequence into the source speech splicing feature module and converting it into a phoneme sequence includes:

[0036] Input a text sequence into the source speech splicing feature module;

[0037] The text sequence is processed using text regularization to form a regular expression;

[0038] The regular expression is converted into Chinese characters through phonetic conversion;

[0039] The Chinese characters are converted into phoneme sequences through polyphonic character classification and prosodic prediction.

[0040] In one embodiment, before the step of inputting the phoneme sequence into the encoder preprocessing network and concatenating the positional encoding, and before the step of inputting the source speech Mel spectrum into the decoder preprocessing network and concatenating the positional encoding features, the method further includes:

[0041] The phoneme sequence is used to perform positional encoding on each frame according to the time sequence, and the formula for positional encoding is as follows:

[0042]

[0043] Where PE is the location code, pos is the location index, i is the dimension index of the time series, and d model For model dimensions.

[0044] On the other hand, a speech conversion device is provided, the device comprising:

[0045] The encoder pre-trained network is used to obtain the source speech language coding features;

[0046] A decoder pre-trained network is used to obtain the source speech fundamental frequency features and concatenate the source speech language coding features with the source speech fundamental frequency features to form source speech concatenation features;

[0047] The target speech encoder is used to obtain the timbre coding features of the target speech.

[0048] A decoder is used to decode the source speech splicing features and the target speech timbre encoding features to obtain the converted timbre Mel spectrum features;

[0049] A vocoder is used to synthesize the converted speech from the converted timbre Mel-spectral features.

[0050] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0051] Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features;

[0052] Obtain the timbre encoding features of the target speech;

[0053] The source speech splicing features and the target speech timbre encoding features are input into the decoder for decoding to obtain the converted timbre Mel spectrum features;

[0054] The converted timbre Mel spectrum features are input into a vocoder to synthesize the converted speech.

[0055] In another aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0056] Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features;

[0057] Obtain the timbre encoding features of the target speech;

[0058] The source speech splicing features and the target speech timbre encoding features are input into the decoder for decoding to obtain the converted timbre Mel spectrum features;

[0059] The converted timbre Mel spectrum features are input into a vocoder to synthesize the converted speech.

[0060] The aforementioned speech conversion method, apparatus, computer equipment, and storage medium concatenate the non-speaker-related language features with the source speaker's fundamental frequency features and input them into a decoder. The target speaker's encoded features are then input into the decoder. The decoder outputs a Mel spectrum with the target speaker's fundamental frequency features. The converted Mel spectrum is then input into a vocoder to synthesize the converted speech. In other words, it employs an end-to-end non-autoregressive model using a transformer and text assistance, introducing multi-head attention to align the non-speaker-related language features of the text and audio. The source speaker's related features and fundamental frequency features are then embedded into the target speaker's language features through the decoder, achieving fast and efficient end-to-end non-autoregressive speech conversion. Therefore, this application can reduce the number of parameters required to convert text to a specific target speaker's speech, shorten the conversion cycle, improve the efficiency of synthesized speech conversion, and reduce the cost of synthesized speech conversion. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating a speech conversion method in one embodiment;

[0063] Figure 2 This is a flowchart illustrating the steps of obtaining source speech language coding features and source speech fundamental frequency features in one embodiment, and concatenating the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features;

[0064] Figure 3 This is a flowchart illustrating the steps of converting a text sequence into a phoneme sequence in one embodiment;

[0065] Figure 4 In one embodiment Figure 1 The diagram shows the module structure corresponding to the speech conversion method shown.

[0066] Figure 5 This is a schematic diagram of the module structure of the non-speaker-related language feature module in one embodiment;

[0067] Figure 6 This is a flowchart illustrating the residual and normalization processing steps in one embodiment.

[0068] Figure 7 This is a flowchart illustrating the processing steps via the forward feedback layer in one embodiment;

[0069] Figure 8 This is a flowchart illustrating the steps of aligning the two-dimensional feature matrix sequence and the decoded spectral features in the multi-head attention layer of the decoder in one embodiment.

[0070] Figure 9 This is a flowchart of the multi-head attention layer of the decoder in one embodiment;

[0071] Figure 10 This is a schematic diagram of the module structure of the multi-head attention layer of the decoder in one embodiment;

[0072] Figure 11 This is a flowchart illustrating the steps of inputting the source speech splicing features after convolution processing and the target speech timbre coding features after conditional batch normalization processing into the batch normalization layer, and splicing and accumulating them with the noise vector in one embodiment.

[0073] Figure 12 This is a structural block diagram of a speech conversion device in one embodiment;

[0074] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0076] like Figure 1 As shown, in one embodiment, a speech conversion method is provided, including the following steps S1-S4.

[0077] Step S1: Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features.

[0078] The source speech language encoding features are source speaker-related language features obtained by converting the text sequence through a non-speaker-related language feature module. The source speech fundamental frequency features are obtained by extracting the source speaker's fundamental frequency features from the source speech Mel spectrum through a fundamental frequency extraction module. Fundamental frequency features include tone and timbre. Fundamental frequency contains a large number of features characterizing speech emotion and is crucial in speech emotion recognition. The source speech language encoding features and source speech fundamental frequency features are concatenated to form source speech concatenation features, i.e., the original text sequence and source speech fundamental frequency features are concatenated to form the initial converted speech, which is usually the machine-default converted timbre speech. This application fuses this machine-default converted timbre speech with the target speaker's timbre speech to achieve timbre conversion; therefore, the target speaker's timbre speech is the pre-imitated timbre speech.

[0079] Mel spectrograms are based on the characteristics of human hearing. We are more sensitive to low-frequency sounds and less sensitive to high-frequency sounds. Therefore, as the frequency of a sound increases linearly, the higher the frequency, the harder it is for us to perceive the difference. Hence, a logarithmic spectrum is used instead of a linear spectrum. Mel spectrograms contain three main characteristics: time-domain and frequency-domain information; perceptually relevant amplitude information; and perceptually relevant frequency-domain information. The core of the Mel spectrum is the Mel-scale, a logarithmic scale that measures linear changes in frequency perception.

[0080] The fundamental frequency extraction module uses the CWT algorithm to extract the fundamental frequency or uses an end-to-end crepe model to extract fundamental frequency features, inputting the Mel spectrum and outputting one-dimensional features.

[0081] Specifically, such as Figure 2 As shown, the step of obtaining the source speech language coding features and the source speech fundamental frequency features, and concatenating the source speech language coding features and the source speech fundamental frequency features to form source speech concatenation features includes S11-S16.

[0082] Step S11: Convert the text sequence into a phoneme sequence.

[0083] The text sequence is the written text, which is the text content to be read aloud. The phoneme sequence is the phonetic sequence of each word and phrase in the text content to be read aloud.

[0084] Among them, such as Figure 3 As shown, the steps for converting a text sequence into a phoneme sequence specifically include: Step S21, inputting the text sequence into the source speech splicing feature module; Step S22, processing the text sequence through text normalization to form a regular expression; Step S23, processing the regular expression through grapheme-to-phoneme conversion to form Chinese; Step S24, converting the Chinese into a phoneme sequence through polyphone classification and prosody prediction.

[0085] Step S12: The phoneme sequence is input into the encoder pre-trained network (encoder prenet), concatenated with the position code, and then input into the encoder multi-head attention layer. After being processed by the residual and normalization layer, the forward feedback layer, the residual and normalization layer, and the linear layer, the text sequence features are output.

[0086] Figure 4 for Figure 1The diagram shows the module structure of the speech conversion method. The speaker encoder, also known as the prenet encoder, uses a GE2E or E-CACPA model, inputting a Mel spectrum (80, 1) and outputting 256-dimensional target speech timbre coding features.

[0087] Figure 5 This is a structural diagram of the non-speaker-related language feature module. The left side of the non-speaker-related language feature module is the encoder PreNet. The encoder PreNet consists of: input text sequence -> 1D convolution -> 1D convolution -> 1D convolution -> (512,). The decoder PreNet consists of: input Mel spectrum -> fully connected layer (256, ReLU) -> fully connected layer (256, ReLU) -> (256,). The values ​​in (512,), (256, ReLU), and (256,) represent the dimensions, and the same applies below.

[0088] Step S13: Input the source speech Mel spectrum into the decoder pre-trained network (decoder prenet), concatenate it with the positional coding features to form a concatenated feature input to the masking multi-head attention layer, and after residual and normalization processing, form the decoded spectral features and input them into the decoder multi-head attention layer.

[0089] like Figure 5 As shown, the right side of the non-speaker-related language feature module is the decoder Prenet. Decoder: Input -> Masked Multi-head Attention (8) -> Residual & Layer Normalization -> Multi-head Attention (8) -> Residual & Layer Normalization -> Forward Feedback Layer -> Residual & Layer Normalization -> Decoder Output.

[0090] like Figure 6 As shown, in this embodiment, the steps of residual and normalization processing include: step S31, input data undergoes first residual and normalization processing; step S32, processed by a forward feedback layer; step S33, undergoes residual and normalization processing; and step S34, outputs synthesized speech after processing by a linear layer.

[0091] like Figure 7 As shown, in this embodiment, the processing steps via the feedforward layer include: step S41, acquiring input data; step S42, inputting the input data into the linear layer; and step S43, the linear layer processing the input data using a piecewise linear function (ReLU activation function) and then outputting the processing result.

[0092] Step S14: The text sequence features and the target speech timbre encoding features are concatenated to form a two-dimensional feature matrix sequence, which is then input into the multi-head attention layer of the decoder.

[0093] The feedforward layer consists of: input -> linear layer -> ReLU activation function -> output.

[0094] Step S15: After the multi-head attention layer of the decoder performs alignment operations on the two-dimensional feature matrix sequence and the decoded spectral features, it outputs the speech of the text to be synthesized after residual and normalization processing.

[0095] Among them, such as Figure 8 , Figure 9 As shown, in this embodiment, the step of aligning the two-dimensional feature matrix sequence and the decoded spectral features in the multi-head attention layer of the decoder includes:

[0096] Step S51: The two-dimensional feature matrix sequence is input into a one-dimensional convolutional layer for convolution processing and then input into the generation module;

[0097] Step S52: Obtain the target speech timbre encoding features, perform conditional batch normalization, and then input them into the generation module;

[0098] Step S53: The generation module inputs the two-dimensional feature matrix sequence after convolution and the target speech timbre coding features after conditional batch normalization into the normalization layer, and concatenates and accumulates them with the noise vector processed by the linear layer.

[0099] Step S54: The output of the generation module is input into a one-dimensional convolutional layer for convolution processing, and then the transformed Mel spectrum features are output.

[0100] An example of the alignment operation steps performed by the multi-head attention layer of the decoder on the two-dimensional feature matrix sequence and the decoded spectral features is as follows:

[0101] 1) The text encoder outputs a text feature sequence after passing through a linear layer, which is then input into the multi-head attention of the decoder as a sequence k,v;

[0102] 2) The decoder's Mel spectrum, after residual and normalization, enters the multi-head attention sequence q;

[0103] 3) Perform SelfAttention alignment calculation on sequences Q (Query), K (Key), and V (Value).

[0104] attention(q,k,v)=softmax(qk_T / sq(d_k))v.

[0105] Step S16: Perform a matrix multiplication between the text sequence features and the speech of the text to be synthesized, and output the source speech splicing features.

[0106] Step S2: Obtain the target speech timbre encoding features.

[0107] Similarly, the method for obtaining the target speech timbre encoding features is to use the target speaker's Mel spectrum and convert it into the target speaker's timbre encoding features through a speaker encoder. The target speaker is the pre-simulated timbre and tone.

[0108] Step S3: Input the source speech splicing features and the target speech timbre encoding features into the decoder for decoding to obtain the converted timbre Mel spectrum features.

[0109] Among them, combined Figure 9 , Figure 10 Step S3 specifically includes: inputting the source speech splicing features into a one-dimensional convolutional layer for convolution processing and then inputting them into the generation module; inputting the target speech timbre encoding features into the generation module after conditional batch normalization processing; inputting the output results of the convolution-processed source speech splicing features and the conditionally batch normalized target speech timbre encoding features processed by the piecewise linear function, upsampling layer, one-dimensional convolutional layer and dilated convolutional layer in the generation module into a one-dimensional convolutional layer for convolution processing and outputting the converted timbre Mel-spectrum features.

[0110] In this embodiment, combined with Figure 10 To simulate the impact of real-world environmental noise, step S3 further includes: processing the noise vector through a linear layer and then inputting it into a batch normalization layer; inputting the source speech splicing features after convolution processing and the target speech timbre encoding features after conditional batch normalization processing into the batch normalization layer, and splicing and accumulating them with the noise vector; inputting the splicing features accumulated with the noise vector into the one-dimensional convolution layer after processing by the piecewise linear function, upsampling layer, one-dimensional convolution layer, and dilated convolution layer in the generation module, performing convolution processing, and outputting the converted timbre Mel spectrum features.

[0111] Combination Figure 10 ,like Figure 11 As shown, the step of inputting the source speech concatenation features after convolution and the target speech timbre coding features after conditional batch normalization into the batch normalization layer, and concatenating and accumulating them with the noise vector, includes:

[0112] Step S61: Use the source speech splicing features after convolution processing and the target speech timbre coding features after conditional batch normalization processing as the first speech input features.

[0113] Step S62: In the generation module, multiple batch normalization layers and multiple piecewise linear functions and upsampling layers are set. The output of one normalization layer is connected to the input of the piecewise linear function and upsampling layer. The output of the piecewise linear function and upsampling layer is connected to the input of a one-dimensional convolutional layer or a dilated convolutional layer. The output of the one-dimensional convolutional layer or the dilated convolutional layer is connected to the input of another normalization layer.

[0114] Step S63: The first speech input feature is input into the batch normalization layer and concatenated with the noise vector. After processing by the piecewise linear function, the upsampling layer, and the one-dimensional convolutional layer, the first concatenated feature is output.

[0115] Step S64: The first speech input feature is processed by a piecewise linear function, an upsampling layer, and a one-dimensional convolutional layer to output the first corrected splicing feature;

[0116] Step S65: Correct the first splicing feature using the first corrected splicing feature to form the second speech input feature;

[0117] Step S66: The second speech input feature is input into the batch normalization layer and concatenated with the noise vector. After processing by the piecewise linear function, upsampling layer, and dilated convolutional layer, the second concatenated feature is output.

[0118] Step S67: The second speech input feature is processed by a piecewise linear function, an upsampling layer, and a dilated convolutional layer to output the second corrected splicing feature;

[0119] Step S68: The second splicing feature is corrected using the second corrected splicing feature to form a speech output feature.

[0120] In this embodiment, before the step of inputting the phoneme sequence into the encoder preprocessing network (prenet) and concatenating it with positional encoding, and before the step of inputting the source speech Mel spectrum into the decoder preprocessing network (prenet) and concatenating it with positional encoding features, the following steps are also included:

[0121] The phoneme sequence is positionally encoded for each frame according to the time sequence. The positional encoding formula is the sin / cosine function, specifically...

[0122]

[0123] Where PE is the location code, pos is the location index, i is the dimension index of the time series, and d model For model dimensions.

[0124] Figure 9 This is a flowchart of the multi-head attention layer (multi-head attention module) of the decoder, where Gblock is the generation module. The generation module mainly consists of one-dimensional convolution, dilated convolution, and a residual network. Simultaneously, a noise vector is injected and concatenated through a linear layer to batch normalize. These modules overlap seven times, as detailed below. Figure 10As shown within the dashed box, the source speaker's relevant features and fundamental frequency features are concatenated and then input into a one-dimensional convolution. The output serves as the input to the generation module. The output of the generation module is then transformed into Mel-spectrum features through a one-dimensional convolution.

[0125] Figure 9 , Figure 10 The corresponding decoder data stream is: spliced ​​features (,) -> one-dimensional convolution (c=512, k=7) -> GBlock (no upsampling, 4X) -> one-dimensional convolution (c=80, k=1) -> Mel spectrum.

[0126] Training process: 1) Alignment training of non-speaker-related language features: The text is processed by the text front-end (text regularization, word segmentation, character-to-phoneme (g2p) conversion, prosody prediction, polyphonic character prediction, etc.) to output a phoneme sequence. The encoder prenet splices the positional encoding information and inputs it into the encoder module (multi-head attention -> residual, normalization -> feedforward layer -> residual, normalization -> linear layer) to output text sequence features. The corresponding Mel spectrum of the text is input into the decoder prenet splices the positional encoding information and inputs it into the decoder module (masking multi-head attention -> residual, normalization multi-head attention -> residual, normalization -> feedforward layer -> residual, normalization -> linear layer) to output a linear layer and multiply it with the encoder output matrix. The encoder output is then input into the multi-head attention layer of the decoder for alignment training, and finally the alignment model is obtained. 2) Overall training: The text is processed by the text front-end (text regularization, word segmentation, character-to-phoneme (g2p) conversion, prosody prediction, polyphonic character prediction, etc.) to output the phoneme sequence. The non-speaker-related feature module outputs the source speaker-related features. The text corresponding to the Mel spectrum is processed by the fundamental frequency extraction module to output the source speaker's fundamental frequency features. The above features are concatenated and input into the decoder. The source speaker's Mel spectrum is input into the speaker encoder and the source speaker's code is input into the decoder. The conversion model is obtained after training by the decoder.

[0127] Application process: The text is processed by a text front-end (text regularization, word segmentation, character-to-phoneme (g2p) conversion, prosody prediction, polyphonic character prediction, etc.) to output a phoneme sequence. The non-speaker-related feature module outputs source speaker-related features. The text's corresponding Mel spectrum is processed by a fundamental frequency extraction module to output the source speaker's fundamental frequency features. The above features are concatenated and input into the decoder. The target speaker's Mel spectrum is input into the speaker encoder, and the target speaker's code is input into the decoder. After decoding, the converted Mel spectrum is output. Finally, the converted speech is synthesized through a traditional or neural vocoder.

[0128] In the aforementioned speech conversion method, the non-speaker-related language features are concatenated with the source speaker's fundamental frequency features and input into the decoder. The target speaker's encoded features are then input into the decoder. The decoder outputs a Mel spectrum with the target speaker's fundamental frequency features. The converted Mel spectrum is then input into a vocoder to synthesize the converted speech. In other words, this method employs an end-to-end non-autoregressive model using a transformer and text assistance, introducing multi-head attention to align the non-speaker-related language features of the text and audio. By embedding the target speaker's language features through the decoder using the source speaker-related features and fundamental frequency features, fast and efficient end-to-end non-autoregressive speech conversion is achieved. Therefore, this application can reduce the number of parameters required to convert text into a specific target speaker's speech, shorten the conversion cycle, improve the efficiency of synthesized speech conversion, and reduce the cost of synthesized speech conversion.

[0129] It should be understood that, although Figures 2-10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-10 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0130] In one embodiment, such as Figure 12 As shown, a speech conversion device 10 is provided, including: an encoder pre-trained network 1, a decoder pre-trained network 2, a target speech encoder 3, a decoder 4, and a vocoder 5. The encoder pre-trained network 1 is used to acquire source speech language coding features; the decoder pre-trained network 2 is used to acquire source speech fundamental frequency features and concatenate the source speech language coding features with the source speech fundamental frequency features to form source speech concatenation features; the target speech encoder 3 is used to acquire target speech timbre coding features; the decoder 4 is used to decode the source speech concatenation features and the target speech timbre coding features to obtain converted timbre Mel spectrum features; and the vocoder 5 is used to synthesize the converted timbre Mel spectrum features into converted speech.

[0131] In the aforementioned speech conversion device, the non-speaker-related language features are concatenated with the source speaker's fundamental frequency features and input into the decoder. The target speaker's encoded features are then input into the decoder. The decoder outputs a Mel spectrum with the target speaker's fundamental frequency features after conversion. The converted Mel spectrum is then input into a vocoder to synthesize the converted speech. That is, by employing an end-to-end non-autoregressive model using a transformer and text assistance, multi-head attention is introduced to align the non-speaker-related language features of the text and audio. The source speaker-related features and fundamental frequency features are then embedded into the target speaker's language features through the decoder, achieving fast and efficient end-to-end non-autoregressive speech conversion. Therefore, this application can reduce the number of parameters required to convert text into a specific target speaker's speech, shorten the conversion cycle, improve the efficiency of synthesized speech conversion, and reduce the cost of synthesized speech conversion.

[0132] For specific limitations regarding the speech conversion device, please refer to the limitations on the speech conversion method above, which will not be repeated here. Each module in the aforementioned speech conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0133] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores speech-to-text data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech-to-text method.

[0134] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech conversion method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0135] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0137] Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features;

[0138] Obtain the timbre encoding features of the target speech;

[0139] The source speech splicing features and the target speech timbre encoding features are input into the decoder for decoding to obtain the converted timbre Mel spectrum features;

[0140] The converted timbre Mel spectrum features are input into a vocoder to synthesize the converted speech.

[0141] For specific limitations on the steps a processor takes when executing a computer program, please refer to the limitations on speech conversion methods mentioned above, which will not be repeated here.

[0142] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0143] Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features;

[0144] Obtain the timbre encoding features of the target speech;

[0145] The source speech splicing features and the target speech timbre encoding features are input into the decoder for decoding to obtain the converted timbre Mel spectrum features;

[0146] The converted timbre Mel spectrum features are input into a vocoder to synthesize the converted speech.

[0147] For specific limitations on the steps implemented when a computer program is executed by a processor, please refer to the limitations on speech conversion methods mentioned above, which will not be repeated here.

[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A speech conversion method, characterized in that, include: Obtain the source speech language coding features and source speech fundamental frequency features, and concatenate the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features; Obtain the timbre encoding features of the target speech; The source speech splicing features and the target speech timbre encoding features are input into the decoder for decoding to obtain the converted timbre Mel spectrum features; The converted timbre Mel-spectral features are input into a vocoder to synthesize the converted speech. The step of obtaining the source speech language coding features and the source speech fundamental frequency features, and concatenating the source speech language coding features and the source speech fundamental frequency features to form source speech concatenation features, includes: Convert a text sequence into a phoneme sequence; The phoneme sequence is input into the encoder pre-trained network, concatenated with the position code, and then input into the encoder multi-head attention layer. After being processed by the residual and normalization layer, the feedforward layer, the residual and normalization layer, and the linear layer, the text sequence features are output. The source speech Mel spectrum is input into the decoder pre-trained network, concatenated with the positional coding features to form a concatenated feature input to the masking multi-head attention layer, and after residual and normalization processing, the decoded spectral features are formed and input into the decoder multi-head attention layer. The text sequence features are concatenated with the target speech timbre encoding features to form a two-dimensional feature matrix sequence, which is then input into the multi-head attention layer of the decoder. The decoder's multi-head attention layer performs alignment operations on the two-dimensional feature matrix sequence and the decoded spectral features, and then outputs the speech of the text to be synthesized after residual and normalization processing. The text sequence features are multiplied by the speech of the text to be synthesized to output the source speech splicing features.

2. The speech conversion method according to claim 1, characterized in that, The step of inputting the source speech splicing features and the target speech timbre encoding features into the decoder for decoding to obtain the converted timbre Mel-spectrum features includes: The source speech splicing features are input into a one-dimensional convolutional layer for convolution processing and then input into the generation module. The target speech timbre coding features are conditionally batch normalized and then input into the generation module. The source speech splicing features after convolution processing and the target speech timbre coding features after conditional batch normalization processing are processed by the piecewise linear function, upsampling layer, one-dimensional convolutional layer and dilated convolutional layer in the generation module. The output is then input into the one-dimensional convolutional layer for convolution processing, and the output is the converted timbre Mel spectrum features.

3. The speech conversion method according to claim 2, characterized in that, The step of inputting the source speech splicing features and the target speech timbre encoding features into the decoder for decoding to obtain the converted timbre Mel spectrum features further includes: The noise vector is processed by a linear layer and then input into the batch normalization layer. The source speech splicing features after convolution processing and the target speech timbre coding features after conditional batch normalization processing are input into the batch normalization layer and spliced ​​and accumulated with the noise vector. The concatenated features, which are concatenated and accumulated with the noise vector, are processed by the piecewise linear function, upsampling layer, one-dimensional convolutional layer, and dilated convolutional layer in the generation module. The output is then input into the one-dimensional convolutional layer for convolution processing, and the converted timbre Mel spectrum features are then output.

4. The speech conversion method according to claim 3, characterized in that, The step of inputting the source speech concatenation features after convolution and the target speech timbre coding features after conditional batch normalization into the batch normalization layer, and concatenating and accumulating them with the noise vector, includes: The source speech splicing features after convolution and the target speech timbre coding features after conditional batch normalization are used as the first speech input features. The generation module is configured with multiple batch normalization layers, multiple piecewise linear functions and upsampling layers. The output of one normalization layer is connected to the input of the piecewise linear function and upsampling layer. The output of the piecewise linear function and upsampling layer is connected to the input of a one-dimensional convolutional layer or a dilated convolutional layer. The output of the one-dimensional convolutional layer or the dilated convolutional layer is connected to the input of another normalization layer. The first speech input feature is input into the batch normalization layer and concatenated with the noise vector and accumulated. After processing by the piecewise linear function, the upsampling layer, and the one-dimensional convolutional layer, the first concatenated feature is output. The first speech input feature is processed by a piecewise linear function, an upsampling layer, and a one-dimensional convolutional layer to output the first corrected spliced ​​feature. The first corrected splicing feature is used to correct the first splicing feature to form the second speech input feature; The second speech input feature is input into the batch normalization layer and concatenated with the noise vector. After processing by the piecewise linear function, upsampling layer, and dilated convolutional layer, the second concatenated feature is output. The second speech input feature is processed by a piecewise linear function, an upsampling layer, and a dilated convolutional layer to output the second corrected splicing feature. The second correction splicing feature is used to correct the second splicing feature to form the speech output feature.

5. The speech conversion method according to claim 1, characterized in that, The step of converting the text sequence into a phoneme sequence includes: Input a text sequence into the source speech splicing feature module; The text sequence is processed using text regularization to form a regular expression; The regular expression is converted into Chinese characters through phonetic conversion; The Chinese characters are converted into phoneme sequences through polyphonic character classification and prosodic prediction.

6. The speech conversion method according to claim 1, characterized in that, Before the step of inputting the phoneme sequence into the encoder preprocessing network and concatenating it with the positional encoding, and before the step of inputting the source speech Mel spectrum into the decoder preprocessing network and concatenating it with the positional encoding features, the method further includes: The phoneme sequence is used to perform positional encoding on each frame according to the time sequence, and the formula for positional encoding is as follows: Where PE is the location code, pos is the location index, i is the dimension index of the time series, and d model For model dimensions.

7. A voice conversion device, characterized in that, The device includes: The encoder pre-trained network is used to obtain the source speech language coding features; A decoder pre-trained network is used to obtain the source speech fundamental frequency features and concatenate the source speech language coding features with the source speech fundamental frequency features to form source speech concatenation features; The target speech encoder is used to obtain the timbre encoding features of the target speech. A decoder is used to decode the source speech splicing features and the target speech timbre encoding features to obtain the converted timbre Mel spectrum features; A vocoder is used to synthesize the converted speech from the converted timbre Mel-spectral features. The step of obtaining source speech language coding features and obtaining source speech fundamental frequency features, and concatenating the source speech language coding features and source speech fundamental frequency features to form source speech concatenation features, includes: Convert a text sequence into a phoneme sequence; The phoneme sequence is input into the encoder pre-trained network, concatenated with the position code, and then input into the encoder multi-head attention layer. After being processed by the residual and normalization layer, the feedforward layer, the residual and normalization layer, and the linear layer, the text sequence features are output. The source speech Mel spectrum is input into the decoder pre-trained network, concatenated with the positional coding features to form a concatenated feature input to the masking multi-head attention layer, and after residual and normalization processing, a decoded spectral feature is formed and input into the decoder multi-head attention layer. The text sequence features are concatenated with the target speech timbre encoding features to form a two-dimensional feature matrix sequence, which is then input into the multi-head attention layer of the decoder. The decoder's multi-head attention layer performs alignment operations on the two-dimensional feature matrix sequence and the decoded spectral features, and then outputs the speech of the text to be synthesized after residual and normalization processing. The text sequence features are multiplied by the speech of the text to be synthesized to output the source speech splicing features.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images