A method and device for converting emotional speech from source emotion to target emotion

Abstract

The present invention relates to the field of speech processing technology, and in particular to a method and device for emotional speech conversion from source emotion to target emotion. The method of the present invention comprises: step one: obtaining speech data Voice1 with source emotion and speech data Voice2 with target emotion; step two: performing feature processing on Voice1 and Voice2 respectively; step three: inputting the result of the processing in step two into a trained emotional speech conversion model for processing to obtain a reconstructed Mel-spectrogram New_Mel3; step four: performing voice encoding restoration on New_Mel3 to obtain speech data Voice3 with target emotion. The emotional speech conversion model designed by the present invention can achieve effective decoupling of emotional information and non-emotional information, and does not need to rely on the target emotional information of the speaker being converted, and can obtain emotional speech with better conversion effect.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and more specifically, to: 1. a method for converting emotional speech from a source emotion to a target emotion; 2. a device for converting emotional speech from a source emotion to a target emotion. Background Art

[0002] Emotional Voice Conversion (EvoC) is a technology that can transform the emotion contained in speech from one category to another. This technology can generate speech with specific emotions or make machine-synthesized speech more emotionally expressive. Compared with traditional speaker voice conversion, emotional voice conversion does not change the speaker information in the speech, but only converts the emotion category of the speech. Given that speech emotions are suprasegmental and hierarchical, and that emotional features are intertwined and difficult to separate from other acoustic features such as speaker identity, current research on emotional voice conversion still faces considerable challenges.

[0003] The inventors are currently working on a scenario where they need to convert a speaker's speech with emotion A into speech with emotion B, but the speaker's emotion B information is missing and difficult or impossible to acquire (i.e., the speaker is not easily able to acquire new emotions). Using existing methods for converting emotion to speech fails to effectively decouple the emotion from the non-emotional information in the speech with emotion A, and also lacks the speaker's emotion B information; both of these factors degrade the conversion performance, resulting in poor results. Summary of the Invention

[0004] Based on this, it is necessary to provide an emotional speech conversion method and device from source emotion to target emotion to address the problem that the existing emotional speech conversion method has poor conversion performance when facing the research scenarios mentioned in the background technology.

[0005] The present invention is achieved by adopting the following technical solutions:

[0006] In a first aspect, the present invention discloses a method for converting emotional speech from a source emotion to a target emotion, comprising:

[0007] Step 1: Obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion;

[0008] Step 2: Extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrogram Mel1;

[0009] Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2;

[0010] Step 3: Input F1, Mel1, and St_Mel2 into the trained emotion speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3;

[0011] Among them, the emotional speech conversion model includes: content encoder, speaker encoder, emotion encoder, decoder, and emotion classifier;

[0012] In step 3, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, and the content encoder extracts the content embedding vector Zc1 from Mel1. The decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3, and the emotion classifier does not work.

[0013] Step 4: Perform voice encoding on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0014] This method of emotional speech conversion from source emotion to target emotion implements the method or process according to an embodiment of the present disclosure.

[0015] In a second aspect, the present invention discloses an emotional speech conversion device from a source emotion to a target emotion, which uses the emotional speech conversion method from a source emotion to a target emotion disclosed in the first aspect.

[0016] The emotional speech conversion device from source emotion to target emotion includes: a speech acquisition module, a feature processing module, a model processing module, and a voice code restoration module.

[0017] The voice acquisition module is used to obtain voice data Voice1 with source emotion and voice data Voice2 with target emotion.

[0018] The feature processing module is used to extract the features of Voice1 and obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 and obtain the original Mel spectrum Mel2; the first-order time domain difference calculation and the second-order time domain difference calculation are performed on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0019] The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3.

[0020] The vocoding restoration module is used to perform vocoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0021] The device for converting emotional speech from a source emotion to a target emotion implements a method or process according to an embodiment of the present disclosure.

[0022] In a third aspect, the present invention discloses a computer program product, comprising a computer program. When the computer program is executed by a processor, the steps of the method for converting emotional speech from a source emotion to a target emotion disclosed in the first aspect are implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The present invention provides a method for emotional speech conversion from source emotion to target emotion. Through the designed emotional speech conversion model, effective decoupling of emotional information and non-emotional information can be achieved, and there is no need to rely on the target emotional information of the speaker being converted, so that emotional speech with better conversion effect can be obtained. After simulation comparison, the converted speech processed by the present invention is superior to the baseline model in both subjective and objective indicators, and the emotion it embodies is closer to the target emotion.

[0025] 2. The emotional speech conversion model of the present invention introduces a separate emotional encoder for learning the emotional information in the reference speech, and uses an emotional classifier to guide the emotional encoder to learn different types of emotional information during model training, thereby ensuring and improving the use effect of the emotional encoder.

[0026] 3. During training, the present invention adjusts the model parameters through a multi-part loss function; among them, the mutual information loss is constructed using the embedding vectors extracted by the three encoders to reduce the coupling between them, so as to ensure that the extracted emotion embedding vector is not mixed with other content; the emotion classification loss is constructed using the results of the emotion classifier to optimize the effect of the emotion decoder; and the reconstruction loss is constructed using two reconstructions to constrain the entire model.

[0027] 4. The present invention does not need to directly use the textual information of the speech in both the training and application stages, which can not only greatly accelerate the training convergence speed of the model, but also shorten the time consumption when the model is applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a structural diagram of the emotional speech conversion model provided in Example 1 of the present invention;

[0030] Figure 2 for Figure 1 The structure diagram of the content encoder in [1].

[0031] Figure 3 for Figure 1 The structure diagram of the speaker encoder in

[15] ;

[0032] Figure 4 for Figure 1 The structure diagram of the decoder in

[15] ;

[0033] Figure 5 for Figure 1 The structure diagram of the emotion encoder in [1].

[0034] Figure 6 for Figure 1 Structure diagram of the sentiment classifier in

[15] ;

[0035] Figure 7 A partial flow chart of training the emotion speech conversion module provided in Example 1 of the present invention;

[0036] Figure 8 A partial flow chart of training the emotion speech conversion module provided in Example 1 of the present invention;

[0037] Figure 9 Flowchart of the emotional speech conversion method from source emotion to target emotion provided by Example 2 of the present invention;

[0038] Figure 10 This is a scatter plot of emotional information provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component. When a component is considered to be "fixed to" another component, it may be directly fixed to the other component or there may be a central component.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0042] Example 1

[0043] This embodiment 1 provides an emotional speech conversion model and discloses its training process.

[0044] See Figure 1 , Figure 1 The structure diagram of the emotional speech conversion model provided in this embodiment 1 is shown, which includes: a content encoder, a speaker encoder, an emotion encoder, a decoder, and an emotion classifier.

[0045] The speaker encoder, content encoder, and decoder can adopt the designs of existing literature (TriAAN-VC method). This embodiment only briefly describes the structure:

[0046] 1. See Figure 2 , the content encoder can be designed as: including: a front convolution part 1, a content convolution part, and a back convolution part 1 connected in sequence.

[0047] The first convolutional part is a BasicConv convolution block.

[0048] The content convolution part includes: 6 content convolution blocks connected in sequence; any content convolution block includes: 1 BasicConv convolution block, 1 BasicConv convolution block, and 1 InstanceNorm layer connected in sequence.

[0049] The post-convolution part is a BasicConv convolution block.

[0050] 2. See Figure 3 , the speaker encoder can be designed as follows: including: the front convolution part 2, the speaker convolution part, and the back convolution part 2 connected in sequence.

[0051] The second front convolution part is a BasicConv convolution block.

[0052] The speaker convolution unit includes: 6 speaker convolution blocks connected in sequence; any speaker convolution block includes: 1 BasicConv convolution block, 1 Speaker-attention layer, and 1 InstanceNorm layer connected in sequence.

[0053] The second post-convolution part is a BasicConv convolution block.

[0054] 3. See Figure 4 The decoder is designed to include: a pre-convolution part, an adaptive part, and a post-processing part connected in sequence.

[0055] The pre-convolution part is a Conv convolution layer.

[0056] The adaptive part includes: 6 adaptive blocks connected in sequence; any adaptive block includes: 1 time adaptation layer, 1 Conv convolution layer, 1 decoder-attention layer, 1 Conv convolution layer, and 1 GRU network layer connected in sequence.

[0057] The post-processing part includes: 1 GRU network layer and 1 post_net layer connected in sequence.

[0058] Among them, the emotion encoder and emotion classifier adopt a new design, which is specifically explained in this embodiment:

[0059] 4. See Figure 5 , the emotion encoder can be designed as follows: including: a two-dimensional convolution part, a recurrent neural part, a fully connected part, and an activation part connected in sequence.

[0060] like Figure 5 As shown in the figure, the two-dimensional convolution part includes: 6 two-dimensional convolution blocks connected in sequence; any two-dimensional convolution block includes: 1 2D convolution layer, 1 BatchNorm layer, and 1 Relu activation function connected in sequence.

[0061] The recurrent neural part includes: a dimension adjustment layer, a GRU network layer, and a dimension squeezing layer connected in sequence.

[0062] The fully connected part consists of two FC layers connected in sequence.

[0063] The activation part is a Relu activation function.

[0064] Overall, the input of the emotion encoder is the Mel spectrum concatenation map (whose dimension is four-dimensional), and the output is the emotion embedding vector (whose dimension is two-dimensional).

[0065] Specifically:

[0066] The two-dimensional convolution part aims to continuously and deeply extract features from the mel spectrum splicing map, and finally obtain a feature extraction map (whose dimension is four-dimensional);

[0067] The dimension adjustment layer performs two rounds of dimension adjustment on the feature extraction map: the first round swaps the second and third dimensions of the feature extraction map to obtain the swap feature map (whose dimension is four-dimensional); the second round multiplies the third and fourth dimensions of the swap feature map into one dimension to obtain the multiplied feature map (whose dimension is three-dimensional);

[0068] The GRU network layer extracts features from the multiplied feature map from the time domain perspective to obtain a time domain feature map (whose dimension is three-dimensional);

[0069] The dimension squeezing layer removes the first dimension of the time domain feature map to obtain a squeezed feature map (whose dimension is two-dimensional);

[0070] The two FC layers in the fully connected part do not change the dimension of the squeezed feature map, thereby obtaining the feature map to be activated (whose dimension is two-dimensional);

[0071] The activation part activates the feature map to be activated, and then obtains the emotion embedding vector.

[0072] 5. See Figure 6 , the sentiment classifier can be designed as follows: including: 3 FC layers connected in sequence.

[0073] Among them, in the sentiment classifier, the first two FC layers do not change the dimension of the sentiment embedding vector; the output dimension of the last FC layer is adjustable - set according to the number of types of sentiment classification required.

[0074] The emotional speech conversion model based on the above structure needs to be trained to obtain a trained emotional speech conversion model so that it can have a good effect when applied.

[0075] This embodiment 1 simultaneously discloses a method for obtaining a trained emotion-speech conversion model, which includes:

[0076] S1, obtain a speech sample set with emotion labels. Generally, an existing well-known dataset can be used, such as the ESD dataset, in which each speech sample contains a correct emotion label.

[0077] S2, divide the speech sample set with emotion labels into training set, validation set, and test set; generally, the ratio of training set, validation set, and test set can be set to 8:1:1.

[0078] S3, performing N rounds of training on the emotional speech conversion model based on the training set, and verifying the emotional speech conversion model after each round of training based on the validation set, until the emotional speech conversion model with the best performance is selected;

[0079] The best performing emotion-speech conversion model is tested on the test set. If it passes the test, it is used as the trained emotion-speech conversion model. Otherwise, S2 is repeated. Note that if S2 is repeated, the training set, validation set, and test set can be adjusted.

[0080] In S3, the training set is divided into N subsets [U1,…,Un,…,UN], and the subset Un is used in the nth round of training; n∈[1,N].

[0081] The process of each round of training is similar, so let’s take the subset Un as an example to illustrate:

[0082] See Figure 7 、 Figure 8 , the methods for using subset Un for the nth round of training include:

[0083] S301, extract the features of Un and obtain the fundamental frequency information F Un , original Mel spectrogram Mel Un .

[0084] Among them, F Un It reflects the pitch information of Un and can ensure the effect of speech reconstruction.

[0085] Among them, Mel Un It reflects the acoustic characteristics of Un, including emotional information, speaker information, and content information.

[0086] To Mel Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf Un , second-order difference graph Δ 2 f Un ; Mel Un , Δf Un , Δ 2 f Un Splice and get the Mel spectrum splicing graph St_Mel Un .

[0087] It should be noted that the emotional information in the speech can be better captured through difference calculation and splicing.

[0088] S302, F Un Mel Un 、St_Mel Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel Un , Classification Result Un .

[0089] Specifically, in S302, the emotion encoder is from St_Mel Un Extract sentiment embedding vector Ze Un , speaker encoder from Mel Un Extract speaker embedding vector Zs Un , content encoder from Mel Un Extract content embedding vector Zc Un , the decoder will Ze Un 、Zs Un 、Zc Un 、F Un Fusion processing into New_Mel Un , the sentiment classifier will Ze Un Perform sentiment classification to obtain Result Un .

[0090] It should be noted that if the speech sample set with emotion labels has M emotion labels, then the last FC layer of the emotion classifier should be configured to classify into M emotions.

[0091] S303, Mel Un Perform data enhancement processing to obtain enhanced Mel-Aug spectrogram Un .

[0092] Among them, data enhancement generally includes: spectrum frequency dimension masking and time dimension masking.

[0093] Mel-Aug Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf-Aug Un , second-order difference graph Δ 2 f-Aug Un ; Mel-Aug Un , Δf-Aug Un , Δ 2 f-Aug Un Perform splicing to obtain the Mel spectrum splicing graph St_Mel_Aug Un .

[0094] S304, F Un 、Mel-Aug Un 、St_Mel_Aug Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel_Aug Un .

[0095] Specifically, in S304, the emotion encoder is from St_Mel_Aug UnExtract sentiment embedding vector Ze_Aug Un , speaker encoder from Mel-Aug Un Extract speaker embedding vector Zs_Aug Un , content encoder from Mel-Aug Un Extract content embedding vector Zc_Aug Un , the decoder is used to convert Ze_Aug Un 、Zs_Aug Un 、Zc_Aug Un 、F Un Fusion processing into New_Mel_Aug Un .

[0096] S305, calculate the loss function Loss of the nth round of training based on S303 and S304 n , and based on Loss n Backpropagation is performed to update the model parameters and obtain the emotional speech conversion model after the nth round of training.

[0097] Specifically, Loss n The expression is:

[0098]

[0099] Where, L_C n represents the sentiment classification loss, C represents the sentiment label, and P(.) represents the probability function.

[0100] L_MI n represents the mutual information loss, Indicates the vCLUB method.

[0101] L_rec n represents the first reconstruction loss; .1 represents the L1 loss function.

[0102] L_siamese n represents the quadratic reconstruction loss.

[0103] In the above loss function calculation process: using L_MI n To reduce the coupling of different embedding vectors, so as to ensure that the extracted emotional embedding vector is not mixed with other content; using L_C n To optimize the effect of the emotion decoder; using L_rec n 、L_siamese n to constrain the entire model.

[0104] Example 2

[0105] See Figure 9 , Figure 9The flowchart of the emotional speech conversion method from source emotion to target emotion provided in this embodiment 2 is shown, which includes:

[0106] Step 1: Obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion.

[0107] It should be noted that Voice1 and Voice2 may belong to different speakers (ie, satisfying the scenario mentioned in the background technology), or may belong to the same speaker (in which case, although this method can be performed, it has little practical significance).

[0108] Step 2: Extract features of Voice1 to obtain fundamental frequency information F1 and original Mel-spectrogram Mel1.

[0109] Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0110] The second step is to perform feature processing on Voice1 and Voice2 respectively, and the processing processes of the two are different.

[0111] Step 3: Input F1, Mel1, and St_Mel2 into the trained emotion speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3.

[0112] Step three is to input the result of step two into the trained emotion-speech conversion model for processing; wherein the trained emotion-speech conversion model is obtained by the method disclosed in Example 1.

[0113] In step three, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, and the content encoder extracts the content embedding vector Zc1 from Mel1. The decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3, and the emotion classifier does not work.

[0114] Step 4: Perform voice encoding on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0115] For Voice3, the sentence content is still consistent with Voice1, but the emotion contained has changed from source emotion to target emotion.

[0116] This embodiment 2 also simultaneously discloses an emotional speech conversion device from a source emotion to a target emotion, which uses the above-mentioned emotional speech conversion method from a source emotion to a target emotion.

[0117] The emotional speech conversion device from source emotion to target emotion includes: a speech acquisition module, a feature processing module, a model processing module, and a voice code restoration module.

[0118] The voice acquisition module is used to obtain voice data Voice1 with source emotion and voice data Voice2 with target emotion.

[0119] The feature processing module is used to extract the features of Voice1 and obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 and obtain the original Mel spectrum Mel2; the first-order time domain difference calculation and the second-order time domain difference calculation are performed on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0120] The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3.

[0121] The vocoding restoration module is used to perform vocoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0122] Since this device uses the above-mentioned emotional speech conversion method from source emotion to target emotion, it also has the same effect and will not be repeated here.

[0123] Simulation Verification

[0124] This embodiment 2 also compares the use effects of the above-mentioned emotional speech conversion method from source emotion to target emotion (abbreviated as Ours) with the CycleGAN-EVC method (abbreviated as CycleGAN-EVC) and the StarGAN-EVC method (abbreviated as StarGAN-EVC).

[0125] The speech sample set with emotion labels uses some English speech data in the ESD dataset. There are four emotion labels: 1. Happy, 2. Sad, 3. Angry, and 4. Neutral.

[0126] First, the emotion embedding vector generated by the emotion encoder is processed using the t-SNE algorithm to form the following Figure 10A scatter plot of .

[0127] Figure 10 Each point in the graph represents a sentence. It can be seen that the sentences with the same emotion can be clustered together well, while there is basically no overlap between the sentences with different emotions. They can be distinguished on the two-dimensional plane: Figure 10 The upper left image shows a speech containing Sad emotion, the upper right image shows a speech containing Happy emotion, the lower left image shows a speech containing Neutral emotion, and the lower right image shows a speech containing Angry emotion. This shows that the emotion encoder has learned the emotional information in the speech data very well. Figure 10 It can be seen that the characteristics of different voices with the same emotion are not completely clustered together, which also reflects the differences in voice styles of the same emotion.

[0128] Next, we tested three transition scenarios: 1. Neutral to Angry; 2. Neutral to Happy; and 3. Neutral to Sad. The three methods were compared using both objective and subjective evaluation metrics. Objective metrics included Mel-Cepstral Distortion (MCD), Perceptual Speech Quality (PESQ), and Root Mean Square Error (FORMSE); while subjective metrics included Mean Opinion Score (MOS) and Best-Worst Scaling (BWS).

[0129] See Table 1, which shows the PESQ comparison of speech data with target emotion obtained by the three methods. The larger the PESQ value, the better.

[0130] Table 1 Comparison of PESQ

[0131]

[0132] It can be seen that the PESQ value of Ours is higher than that of the other two baseline methods, indicating that the speech data with target emotion converted by Ours has better auditory perception quality.

[0133] See Table 2, which shows the MCD comparison of speech data with target emotion obtained by the three methods. The smaller the MCD value, the better.

[0134] Table 2 Comparison of MCD

[0135]

[0136] It can be seen that Ours obtains a lower MCD than the other two methods, indicating that the speech data with target emotion converted by Ours has a spectrum closer to the real speech data with target emotion.

[0137] See Table 3, which shows the comparison of the F0RMSE of speech data with target emotion obtained by the three methods. The smaller the F0RMSE value, the better.

[0138] Table 3 Comparison of F0RMSE

[0139]

[0140] It can be seen that Ours obtains a lower F0RMSE value than the other two methods, indicating that Ours has a better sentiment conversion effect.

[0141] See Table 4, which shows the MOS comparison of speech data with target emotion obtained by the three methods. The larger the MOS value, the better.

[0142] Table 4 MOS comparison

[0143]

[0144] It can be seen that Ours obtains a higher MOS score than the other two methods, indicating that Ours can better achieve high-quality conversion of speech emotions.

[0145] See Table 5, which shows the BWS comparison of speech data with target emotion obtained by the three methods. The larger the BWS value, the better.

[0146] Table 5 Comparison of BWS

[0147]

[0148] It can be seen that Ours obtains better BWS evaluation results than the other two methods, indicating that the emotional accuracy, richness and voice quality of the speech data with target emotion converted by Ours are better than those of the other two methods.

[0149] Combining the above objective evaluation indicators and subjective evaluation indicators, the effectiveness of the above emotional speech conversion method from source emotion to target emotion is verified.

[0150] Example 3

[0151] This embodiment 3 discloses a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method for converting emotional speech from source emotion to target emotion disclosed in embodiment 1.

[0152] This embodiment 3 also discloses a readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the emotional speech conversion method from source emotion to target emotion disclosed in embodiment 1 are executed.

[0153] This embodiment 3 further discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for converting emotional speech from a source emotion to a target emotion disclosed in embodiment 1 are implemented.

[0154] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for converting emotional speech from source emotion to target emotion, characterized in that: include: Step 1: Obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion; Step 2: Extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrogram Mel1; Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2; Step 3: Input F1, Mel1, and St_Mel2 into the trained emotion speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3; Among them, the emotional speech conversion model includes: content encoder, speaker encoder, emotion encoder, decoder, and emotion classifier; In step 3, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, and the content encoder extracts the content embedding vector Zc1 from Mel1. The decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3, and the emotion classifier does not work. Step 4: Perform voice encoding on New_Mel3 to obtain the speech data Voice3 with the target emotion.

2. The method for converting emotional speech from source emotion to target emotion according to claim 1, wherein: Voice1 and Voice2 belong to different speakers or the same speaker.

3. The method for converting emotional speech from source emotion to target emotion according to claim 1, wherein: Methods for obtaining a trained emotional speech conversion model include: S1, obtain a speech sample set with emotion labels; S2, divide the speech sample set with emotion labels into training set, validation set, and test set; S3, performing N rounds of training on the emotional speech conversion model based on the training set, and verifying the emotional speech conversion model after each round of training based on the validation set, until the emotional speech conversion model with the best performance is selected; The best-performing emotion-speech conversion model is tested based on the test set; if the test passes, the best-performing emotion-speech conversion model is used as the trained emotion-speech conversion model; otherwise, S2 is repeated.

4. The method for converting emotional speech from source emotion to target emotion according to claim 3, wherein: In S3, the training set is divided into N subsets [U1,…,Un,…,UN], and the subset Un is used in the nth round of training; n∈[1,N]; Among them, the method of using the subset Un for the nth round of training includes: S301, extract the features of Un and obtain the fundamental frequency information F Un , original Mel spectrogram Mel Un ; To Mel Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf Un , second-order difference graph Δ 2 f Un ; Mel Un , Δf Un , Δ 2 f Un Perform splicing to obtain the Mel spectrum splicing graph St_Mel Un ; S302, F Un Mel Un 、St_Mel Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel Un , Classification Result Un ; S303, Mel Un Perform data enhancement processing to obtain enhanced Mel-Aug spectrogram Un ; Mel-Aug Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf-Aug Un , second-order difference graph Δ 2 f-Aug Un ; Mel-Aug Un , Δf-Aug Un , Δ 2 f-Aug Un Perform splicing to obtain the Mel spectrum splicing graph St_Mel_Aug Un ; S304, F Un 、Mel-Aug Un 、St_Mel_Aug Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel_Aug Un ; S305, calculate the loss function Loss of the nth round of training based on S303 and S304 n , and based on Loss n Backpropagation is performed to update the model parameters and obtain the emotional speech conversion model after the nth round of training.

5. The method for converting emotional speech from source emotion to target emotion according to claim 4, wherein: In S302, the emotion encoder is Un Extract sentiment embedding vector Ze Un , speaker encoder from Mel Un Extract speaker embedding vector Zs Un , content encoder from Mel Un Extract content embedding vector Zc Un , the decoder will Ze Un 、Zs Un 、Zc Un 、F Un Fusion processing into New_Mel Un , the sentiment classifier will Ze Un Perform sentiment classification to obtain Result Un ; In S304, the emotion encoder is Un Extract sentiment embedding vector Ze_Aug Un , speaker encoder from Mel-Aug Un Extract speaker embedding vector Zs_Aug Un , content encoder from Mel-Aug Un Extract content embedding vector Zc_Aug Un , the decoder is used to convert Ze_Aug Un 、Zs_Aug Un 、Zc_Aug Un 、F Un Fusion processing into New_Mel_Aug Un .

6. The method for converting emotional speech from source emotion to target emotion according to claim 5, wherein: Loss n The expression is: Where, L_C n represents sentiment classification loss, C represents sentiment label, and P(.) represents probability function; L_MI n represents the mutual information loss, Represents the vCLUB method; L_rec n represents the first reconstruction loss; ||.||1 represents the L1 loss function; L_siamese n represents the quadratic reconstruction loss.

7. The method for converting emotional speech from source emotion to target emotion according to claim 1, wherein: The emotion encoder includes: a two-dimensional convolution part, a recurrent neural part, a fully connected part, and an activation part connected in sequence; The two-dimensional convolution part includes: 6 two-dimensional convolution blocks connected in sequence; any two-dimensional convolution block includes: 1 2D convolution layer, 1 BatchNorm layer, and 1 Relu activation function connected in sequence; The recurrent neural network consists of a dimension adjustment layer, a GRU network layer, and a dimension squeezing layer connected in sequence. The fully connected part consists of two FC layers connected in sequence; The activation part is a Relu activation function.

8. The method for converting emotional speech from source emotion to target emotion according to claim 1, wherein: The sentiment classifier includes: three FC layers connected in sequence.

9. An emotional speech conversion device from source emotion to target emotion, characterized in that: It uses the emotional speech conversion method from source emotion to target emotion as described in any one of claims 1-8; The emotional speech conversion device from source emotion to target emotion includes: The voice acquisition module is used to obtain the voice data Voice1 with source emotion and the voice data Voice2 with target emotion; The feature processing module is used to extract the features of Voice1 and obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 and obtain the original Mel spectrum Mel2; it performs the first-order time domain difference calculation and the second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2; The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotion speech conversion model for processing to obtain the reconstructed mel spectrogram New_Mel3; as well as The voice encoding restoration module is used to perform voice encoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for converting emotional speech from a source emotion to a target emotion as described in any one of claims 1 to 8 are implemented.

Images