Emotional voice conversion method and device from source emotion to target emotion

Abstract

The invention relates to the technical field of voice processing, in particular to an emotion voice conversion method and device from source emotion to target emotion. The method comprises the following steps of: 1, acquiring voice data Voice1 with source emotion and voice data Voice2 with target emotion; step 2, carrying out feature processing on the Voice1 and the Voice2 respectively; step 3, inputting a processing result of the step 2 into a trained emotional speech conversion model for processing to obtain a reconstructed Mel spectrum NewMel3; and step 4, carrying out sound code reduction on the NewMel3 to obtain voice data Voice3 with target emotion. Through the designed emotional voice conversion model, effective decoupling of emotional information and non-emotional information can be realized, dependence on target emotional information of a converted speaker is not needed, and emotional voice with a better conversion effect can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and more specifically, to: 1. a method for converting an emotion speech from a source emotion to a target emotion; 2. a device for converting an emotion speech from a source emotion to a target emotion. Background Art

[0002] Emotional Voice Conversion is a technology that can transform the emotion contained in speech from one category to another. This technology can generate speech with specific emotions, or make machine-synthesized speech have richer emotional expression. Compared with traditional speaker voice conversion, emotional voice conversion does not change the speaker information in the speech, but only converts the emotion category of the speech. Given that speech emotions are suprasegmental and hierarchical, and that emotional features are intertwined with other acoustic features such as speaker identity and difficult to separate, current research on emotional voice conversion still faces considerable challenges.

[0003] The research scenario currently encountered by the inventor is: it is necessary to convert a speaker's speech with emotion A into speech with emotion B, but the speaker's emotion B information is missing and difficult or impossible to obtain (that is, it is difficult for the speaker to acquire new emotions). If the existing emotional speech conversion method is used, on the one hand, the emotional information and non-emotional information of the speech with emotion A cannot be effectively decoupled, and on the other hand, the speaker's emotion B information is missing; both aspects will degrade the conversion performance, resulting in poor conversion performance. Summary of the invention

[0004] Based on this, it is necessary to provide an emotional speech conversion method and device from source emotion to target emotion to address the problem that the existing emotional speech conversion methods have poor conversion performance when facing the research scenarios mentioned in the background technology.

[0005] The present invention is implemented by the following technical solutions:

[0006] In a first aspect, the present invention discloses a method for converting emotional speech from a source emotion to a target emotion, comprising:

[0007] Step 1: obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion;

[0008] Step 2: Extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrogram Mel1;

[0009] Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2;

[0010] Step 3: Input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed Mel spectrogram New_Mel3;

[0011] Among them, the emotional speech conversion model includes: content encoder, speaker encoder, emotion encoder, decoder, emotion classifier;

[0012] In step 3, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, the content encoder extracts the content embedding vector Zc1 from Mel1, and the decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3. The emotion classifier does not work.

[0013] Step 4: Perform voice encoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0014] This method of emotional speech conversion from source emotion to target emotion implements the method or process according to an embodiment of the present disclosure.

[0015] In a second aspect, the present invention discloses an emotional speech conversion device from a source emotion to a target emotion, which uses the emotional speech conversion method from a source emotion to a target emotion disclosed in the first aspect.

[0016] The emotional speech conversion device from source emotion to target emotion includes: a speech acquisition module, a feature processing module, a model processing module, and a voice code restoration module.

[0017] The voice acquisition module is used to obtain voice data Voice1 with source emotion and voice data Voice2 with target emotion.

[0018] The feature processing module is used to extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 to obtain the original Mel spectrum Mel2; the first-order time domain difference calculation and the second-order time domain difference calculation are performed on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0019] The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed mel-spectrogram New_Mel3.

[0020] The voice code restoration module is used to perform voice code restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0021] The device for converting emotional speech from source emotion to target emotion implements the method or process according to the embodiment of the present disclosure.

[0022] In a third aspect, the present invention discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for converting emotional speech from a source emotion to a target emotion disclosed in the first aspect are implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The present invention provides an emotional speech conversion method from source emotion to target emotion. Through the designed emotional speech conversion model, effective decoupling of emotional information and non-emotional information can be achieved, and there is no need to rely on the target emotional information of the speaker being converted, so that emotional speech with better conversion effect can be obtained. After simulation comparison, the converted speech processed by the present invention is superior to the baseline model in both subjective and objective indicators, and the emotion it embodies is closer to the target emotion.

[0025] 2. The emotional speech conversion model of the present invention introduces a separate emotional encoder for learning the emotional information in the reference speech, and uses an emotional classifier to guide the emotional encoder to learn different types of emotional information during model training, thereby ensuring and improving the use effect of the emotional encoder.

[0026] 3. The present invention adjusts the parameters of the model through a multi-part loss function during training; wherein, the mutual information loss is constructed using the embedding vectors extracted by the three encoders to reduce the coupling between them, so as to ensure that the extracted emotion embedding vector is not mixed with other contents; the emotion classification loss is constructed using the result of the emotion classifier to optimize the effect of the emotion decoder; and the reconstruction loss is constructed using two reconstructions to constrain the entire model.

[0027] 4. The present invention does not need to directly use the textual quotation information of the speech in both the training and application stages, which can not only greatly accelerate the training convergence speed of the model, but also shorten the time consumption of the model application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A structural diagram of the emotional speech conversion model provided in Example 1 of the present invention;

[0030] Figure 2 for Figure 1 Structure diagram of the content encoder in ;

[0031] Figure 3 for Figure 1 Structural diagram of the speaker encoder in [1].

[0032] Figure 4 for Figure 1 The structure diagram of the decoder in the figure;

[0033] Figure 5 for Figure 1 Structure diagram of the emotion encoder in the figure;

[0034] Figure 6 for Figure 1 Structure diagram of the sentiment classifier in the figure;

[0035] Figure 7 A partial flow chart of training the emotional speech conversion module provided in Example 1 of the present invention;

[0036] Figure 8 A partial flow chart of training the emotional speech conversion module provided in Example 1 of the present invention;

[0037] Fig. 9 A flowchart of a method for converting an emotional speech from a source emotion to a target emotion provided in Embodiment 2 of the present invention;

[0038] Fig.10 This is a scatter plot of emotional information provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component at the same time. When a component is considered to be "fixed to" another component, it may be directly fixed on the other component or there may be a central component at the same time.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0042] Example 1

[0043] This embodiment 1 provides an emotional speech conversion model and discloses its training process.

[0044] See also Figure 1 , Figure 1 The structure diagram of the emotional speech conversion model provided in this embodiment 1 is shown, which includes: a content encoder, a speaker encoder, an emotion encoder, a decoder, and an emotion classifier.

[0045] Among them, the speaker encoder, content encoder, and decoder can adopt the design of the existing literature (TriAAN-VC method). This embodiment only briefly describes the structure:

[0046] 1. See Figure 2 , the content encoder can be designed as: including: a front convolution part 1, a content convolution part, and a back convolution part 1 connected in sequence.

[0047] The first convolutional part is a BasicConv convolution block.

[0048] The content convolution part includes: 6 content convolution blocks connected in sequence; any content convolution block includes: 1 BasicConv convolution block, 1 BasicConv convolution block, and 1 InstanceNorm layer connected in sequence.

[0049] The post-convolution part is a BasicConv convolution block.

[0050] 2. See Figure 3 , the speaker encoder can be designed as: including: a front convolution part 2, a speaker convolution part, and a back convolution part 2 connected in sequence.

[0051] The second front convolution part is a BasicConv convolution block.

[0052] The speaker convolution part includes: 6 speaker convolution blocks connected in sequence; any speaker convolution block includes: 1 BasicConv convolution block, 1 Speaker-attention layer, and 1 InstanceNorm layer connected in sequence.

[0053] The second post-convolution part is a BasicConv convolution block.

[0054] 3. See Figure 4 The decoder is designed to include: a pre-convolution part, an adaptive part, and a post-processing part connected in sequence.

[0055] The pre-convolution part is 1 Conv convolution layer.

[0056] The adaptive part includes: 6 adaptive blocks connected in sequence; any adaptive block includes: 1 Time adaptive layer, 1 Conv convolution layer, 1 decoder-attention layer, 1 Conv convolution layer, and 1 GRU network layer connected in sequence.

[0057] The post-processing part includes: 1 GRU network layer and 1 post_net layer connected in sequence.

[0058] Among them, the emotion encoder and emotion classifier adopt a new design, which is specifically described in this embodiment:

[0059] 4. See Figure 5 , the emotion encoder can be designed as follows: including: a two-dimensional convolution part, a recurrent neural part, a fully connected part, and an activation part connected in sequence.

[0060] like Figure 5 As shown, the two-dimensional convolution part includes: 6 two-dimensional convolution blocks connected in sequence; any two-dimensional convolution block includes: 1 2D convolution layer, 1 BatchNorm layer, and 1 Relu activation function connected in sequence.

[0061] The recurrent neural part includes: a dimension adjustment layer, a GRU network layer, and a dimension squeezing layer connected in sequence.

[0062] The fully connected part includes two FC layers connected in sequence.

[0063] The activation part is a Relu activation function.

[0064] Overall, the input of the emotion encoder is the Mel spectrum concatenation map (whose dimension is four dimensions), and the output is the emotion embedding vector (whose dimension is two dimensions).

[0065] Specifically:

[0066] The two-dimensional convolution part aims to continuously and deeply extract features from the mel spectrum concatenation map, and finally obtain a feature extraction map (whose dimension is four-dimensional);

[0067] The dimension adjustment layer performs two rounds of dimension adjustment on the feature extraction map: in the first round, the second and third dimensions of the feature extraction map are swapped to obtain an interchanged feature map (whose dimension is four-dimensional); in the second round, the third and fourth dimensions of the interchanged feature map are multiplied into one dimension to obtain a multiplied feature map (whose dimension is three-dimensional);

[0068] The GRU network layer extracts features from the multiplied feature map from the time domain perspective to obtain a time domain feature map (whose dimension is three-dimensional);

[0069] The dimension squeezing layer removes the first dimension of the time domain feature map to obtain a squeezed feature map (whose dimension is two-dimensional);

[0070] The two FC layers in the fully connected part do not change the dimension of the squeezed feature map, thereby obtaining a feature map to be activated (whose dimension is two-dimensional);

[0071] The activation unit activates the feature map to be activated, and then obtains the sentiment embedding vector.

[0072] 5. See Figure 6 , the sentiment classifier can be designed as follows: including: 3 FC layers connected in sequence.

[0073] Among them, in the sentiment classifier, the first two FC layers do not change the dimension of the sentiment embedding vector; the output dimension of the last FC layer is adjustable - set according to the number of types of sentiment classification required.

[0074] The emotional speech conversion model based on the above structure needs to be trained to obtain a trained emotional speech conversion model so that it can have a good use effect when applied.

[0075] This embodiment 1 simultaneously discloses a method for obtaining a trained emotion speech conversion model, which includes:

[0076] S1, obtain a speech sample set with emotion labels. Generally, an existing well-known data set can be used, such as the ESD data set, in which each speech contains a correct emotion label.

[0077] S2, divide the speech sample set with emotion labels into training set, validation set, and test set; generally, the ratio of training set, validation set, and test set can be set to 8:1:1.

[0078] S3, performing N rounds of training on the emotional speech conversion model based on the training set, and verifying the emotional speech conversion model after each round of training based on the verification set, until the emotional speech conversion model with the best performance is selected;

[0079] The best performing emotion speech conversion model is tested based on the test set; if the test passes, the best performing emotion speech conversion model is used as the trained emotion speech conversion model; otherwise, S2 is repeated. It should be noted that if S2 is repeated, the training set, validation set, and test set can be adjusted.

[0080] In S3, the training set is divided into N subsets [U1,…,Un,…,UN], and the nth round of training uses the subset Un; n∈[1,N].

[0081] The process of each round of training is similar, so let's take the subset Un as an example to illustrate:

[0082] See also Figure 7 , Figure 8 , the method of using subset Un for the nth round of training includes:

[0083] S301, extract features from Un to obtain baseband information F Un , original Mel spectrogram Mel Un .

[0084] Among them, F Un It reflects the pitch information of Un and can ensure the effect of speech reconstruction.

[0085] Among them, Mel Un It reflects the acoustic characteristics of Un, including emotional information, speaker information, and content information.

[0086] To Mel Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf Un , second-order difference graph Δ 2 f Un ; Mel Un , Δf Un , Δ 2 f Un Stitching is performed to obtain the Mel spectrum stitching graph St_Mel Un .

[0087] It should be noted that the emotional information in the speech can be better captured through difference calculation and splicing.

[0088] S302, F Un Mel Un , St_Mel Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel Un , Classification Result Un .

[0089] Specifically, in S302, the emotion encoder is from St_Mel Un Extract sentiment embedding vector Ze Un , the speaker encoder is from Mel Un Extract speaker embedding vector Zs Un , content encoder from Mel Un Extract content embedding vector Zc Un , the decoder will Ze Un 、Zs Un , Zc Un 、F Un Fusion processing into New_Mel Un , the sentiment classifier will be Ze Un Perform sentiment classification to get Result Un .

[0090] It should be noted that if the speech sample set with emotion labels has M emotion labels, then the last FC layer of the emotion classifier should be configured to classify into M emotions.

[0091] S303, Mel Un Perform data enhancement processing to obtain enhanced Mel-Aug spectrogram Un .

[0092] Among them, data enhancement generally includes: spectral frequency dimension masking and time dimension masking.

[0093] Mel-Aug Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf-Aug Un , second-order difference graph Δ 2 f-Aug Un ; Mel-Aug Un , Δf-Aug Un , Δ 2 f-Aug Un Stitching is performed to obtain the Mel spectrum stitching graph St_Mel_Aug Un .

[0094] S304, F Un , Mel-Aug Un 、St_Mel_Aug Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel_Aug Un .

[0095] Specifically, in S304, the emotion encoder is from St_Mel_Aug UnExtract sentiment embedding vector Ze_Aug Un , speaker encoder from Mel-Aug Un Extract speaker embedding vector Zs_Aug Un , content encoder from Mel-Aug Un Extract content embedding vector Zc_Aug Un , the decoder is used to convert Ze_Aug Un 、Zs_Aug Un 、Zc_Aug Un 、F Un Fusion processing into New_Mel_Aug Un .

[0096] S305, based on S303 and S304, calculate the loss function Loss of the nth round of training n , and based on Loss n Back propagation is performed to update the model parameters and obtain the emotional speech conversion model after the nth round of training.

[0097] Specifically, Loss n The expression is:

[0098]

[0099] In the formula, L_C n represents the sentiment classification loss, C represents the sentiment label, and P(.) represents the probability function.

[0100] L_MI n represents the mutual information loss, Indicates the vCLUB method.

[0101] L_rec n represents the first reconstruction loss; .1 represents the L1 loss function.

[0102] L_siamese n represents the secondary reconstruction loss.

[0103] In the above loss function calculation process: using L_MI n To reduce the coupling of different embedding vectors, to ensure that the extracted emotional embedding vector is not mixed with other content; using L_C n To optimize the effect of the emotion decoder; using L_rec n , L_siamese n to constrain the entire model.

[0104] Example 2

[0105] See also Fig. 9 , Fig. 9A flowchart of the method for converting the emotional speech from the source emotion to the target emotion provided in this embodiment 2 is shown, which includes:

[0106] Step 1: Obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion.

[0107] It should be noted that Voice1 and Voice2 may belong to different speakers (ie, satisfying the scenario mentioned in the background technology), or may belong to the same speaker (in this case, although the method can be performed, it has little practical significance).

[0108] Step 2: Extract features of Voice1 to obtain fundamental frequency information F1 and original Mel-spectrogram Mel1.

[0109] Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0110] The second step is to perform feature processing on Voice1 and Voice2 respectively, and the processing processes of the two are different.

[0111] Step 3: Input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed mel-spectrogram New_Mel3.

[0112] Step three is to input the result of step two into the trained emotion speech conversion model for processing; wherein the trained emotion speech conversion model is obtained by the method disclosed in Example 1.

[0113] In step three, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, the content encoder extracts the content embedding vector Zc1 from Mel1, and the decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3. The emotion classifier does not work.

[0114] Step 4: Perform voice encoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0115] For Voice3, the sentence content is still consistent with Voice1, but the emotion contained has changed from source emotion to target emotion.

[0116] This embodiment 2 also simultaneously discloses an emotional speech conversion device from a source emotion to a target emotion, which uses the above-mentioned emotional speech conversion method from a source emotion to a target emotion.

[0117] The emotional speech conversion device from source emotion to target emotion includes: a speech acquisition module, a feature processing module, a model processing module, and a voice code restoration module.

[0118] The voice acquisition module is used to obtain voice data Voice1 with source emotion and voice data Voice2 with target emotion.

[0119] The feature processing module is used to extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 to obtain the original Mel spectrum Mel2; the first-order time domain difference calculation and the second-order time domain difference calculation are performed on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2.

[0120] The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed mel-spectrogram New_Mel3.

[0121] The voice code restoration module is used to perform voice code restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

[0122] Since the present device uses the above-mentioned emotional speech conversion method from source emotion to target emotion, it also has the same effect and will not be repeated here.

[0123] Simulation Verification

[0124] This embodiment 2 also compares the use effects of the above-mentioned emotional speech conversion method from source emotion to target emotion (abbreviated as Ours) with the CycleGAN-EVC method (abbreviated as CycleGAN-EVC) and the StarGAN-EVC method (abbreviated as StarGAN-EVC).

[0125] The speech sample set with emotion labels uses part of the English speech data in the ESD dataset. There are four emotion labels: 1. Happy, 2. Sad, 3. Angry, and 4. Neutral.

[0126] First, the sentiment embedding vector generated by the sentiment encoder is processed using the t-SNE algorithm to form the following Fig.10A scatter plot of .

[0127] Fig.10 Each point in the graph represents a sentence. It can be seen that the speech with the same emotion can be well clustered together, and there is basically no overlap between the speech with different emotions. They can be distinguished on the two-dimensional plane: Fig.10 The upper left image shows the speech with Sad emotion, the upper right image shows the speech with Happy emotion, the lower left image shows the speech with Neutral emotion, and the lower right image shows the speech with Angry emotion. This shows that the emotion encoder has learned the emotion information in the speech data very well. Fig.10 It can be seen that the characteristics of different voices with the same emotion are not completely gathered together, which also reflects the differences in voice styles of the same emotion.

[0128] Secondly, three conversion scenarios were tested: 1. Neutral to angry; 2. Neutral to happy; 3. Neutral to sad. The comparison of the three methods was quantified using objective evaluation indicators and subjective evaluation indicators. Among them, the objective evaluation indicators include: Mel-Cepstral Distortion (MCD), Perceptual Speech Quality Evaluation (PESQ), and Root Mean Square Error (FORMSE); the subjective evaluation indicators include: Mean Opinion Score (MOS) and Best-Worst Scaling (BWS).

[0129] See Table 1, which shows the PESQ comparison of speech data with target emotion obtained by three methods. The larger the PESQ value, the better.

[0130] Table 1 Comparison of PESQ

[0131]

[0132] It can be seen that the PESQ value of Ours is higher than that of the other two baseline methods, indicating that the speech data with target emotion converted by Ours has better auditory perception quality.

[0133] See Table 2, which shows the MCD comparison of speech data with target emotion obtained by three methods. The smaller the MCD value, the better.

[0134] Table 2 Comparison of MCD

[0135]

[0136] It can be seen that Ours obtains a lower MCD than the other two methods, indicating that the speech data with target emotion converted by Ours has a spectrum closer to the real speech data with target emotion.

[0137] See Table 3, which shows the comparison of F0RMSE of speech data with target emotion obtained by three methods. The smaller the F0RMSE value, the better.

[0138] Table 3 Comparison of F0RMSE

[0139]

[0140] It can be seen that Ours obtains a lower F0RMSE value than the other two methods, indicating that Ours has a better sentiment conversion effect.

[0141] See Table 4, which shows the MOS comparison of speech data with target emotions obtained by the three methods. The larger the MOS value, the better.

[0142] Table 4 MOS comparison

[0143]

[0144] It can be seen that Ours obtains a higher MOS score than the other two methods, indicating that Ours can better achieve high-quality conversion of speech emotions.

[0145] See Table 5, which shows the BWS comparison of speech data with target emotion obtained by three methods. The larger the BWS value, the better.

[0146] Table 5 Comparison of BWS

[0147]

[0148] It can be seen that Ours obtains better BWS evaluation results than the other two methods, indicating that the emotional accuracy, richness and voice quality of the speech data with target emotion converted by Ours are better than those of the other two methods.

[0149] Combining the above objective evaluation indicators and subjective evaluation indicators, the effectiveness of the above emotional speech conversion method from source emotion to target emotion is verified.

[0150] Example 3

[0151] This embodiment 3 discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method for converting an emotion speech from a source emotion to a target emotion disclosed in embodiment 1 are implemented.

[0152] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of the method for converting the emotion speech from the source emotion to the target emotion disclosed in embodiment 1 are executed.

[0153] This embodiment 3 also discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method for converting the emotional speech from the source emotion to the target emotion disclosed in embodiment 1 are implemented.

[0154] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for converting emotional speech from source emotion to target emotion, characterized in that: include: Step 1: obtain the speech data Voice1 with source emotion and the speech data Voice2 with target emotion; Step 2: Extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrogram Mel1; Perform feature extraction on Voice2 to obtain the original Mel spectrum Mel2; perform first-order time domain difference calculation and second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2; Step 3: Input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed Mel spectrogram New_Mel3; Among them, the emotional speech conversion model includes: content encoder, speaker encoder, emotion encoder, decoder, emotion classifier; In step 3, the emotion encoder extracts the emotion embedding vector Ze2 from St_Mel2, the speaker encoder extracts the speaker embedding vector Zs1 from Mel1, the content encoder extracts the content embedding vector Zc1 from Mel1, and the decoder fuses Ze2, Zs1, Zc1, and F1 into New_Mel3. The emotion classifier does not work. Step 4: Perform voice encoding restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

2. The method for converting emotional speech from source emotion to target emotion according to claim 1, characterized in that: Voice1 and Voice2 belong to different speakers or the same speaker.

3. The method for converting emotional speech from source emotion to target emotion according to claim 1, characterized in that: Methods for obtaining the trained emotional speech conversion model include: S1, obtain a speech sample set with emotion labels; S2, divide the speech sample set with emotion labels into training set, validation set and test set; S3, performing N rounds of training on the emotional speech conversion model based on the training set, and verifying the emotional speech conversion model after each round of training based on the verification set, until the emotional speech conversion model with the best performance is selected; The best performing emotion speech conversion model is tested based on the test set; if the test passes, the best performing emotion speech conversion model is used as the trained emotion speech conversion model; otherwise, S2 is repeated.

4. The method for converting emotional speech from source emotion to target emotion according to claim 3, characterized in that: In S3, the training set is divided into N subsets [U1,…,Un,…,UN], and the subset Un is used in the nth round of training; n∈[1,N]; Among them, the method of using the subset Un for the nth round of training includes: S301, extract features from Un to obtain baseband information F Un , original Mel spectrogram Mel Un ; To Mel Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf Un , second-order difference graph Δ 2 f Un ; Mel Un , Δf Un , Δ 2 f Un Stitching is performed to obtain the Mel spectrum stitching graph St_Mel Un ; S302, F Un Mel Un , St_Mel Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel Un , Classification Result Un ; S303, Mel Un Perform data enhancement processing to obtain enhanced Mel-Aug spectrogram Un ; Mel-Aug Un Perform the first-order time domain difference calculation and the second-order time domain difference calculation in sequence to obtain the first-order difference graph Δf-Aug Un , second-order difference graph Δ 2 f-Aug Un ; Mel-Aug Un , Δf-Aug Un , Δ 2 f-Aug Un Stitching is performed to obtain the Mel spectrum stitching graph St_Mel_Aug Un ; S304, F Un , Mel-Aug Un 、St_Mel_Aug Un Input the emotional speech conversion model that has been trained for the n-1th round for processing to obtain the reconstructed Mel spectrogram New_Mel_Aug Un ; S305, based on S303 and S304, calculate the loss function Loss of the nth round of training n , and based on Loss n Back propagation is performed to update the model parameters and obtain the emotional speech conversion model after the nth round of training.

5. The method for converting emotional speech from source emotion to target emotion according to claim 4, characterized in that: In S302, the emotion encoder is from St_Mel Un Extract sentiment embedding vector Ze Un , the speaker encoder is from Mel Un Extract speaker embedding vector Zs Un , content encoder from Mel Un Extract content embedding vector Zc Un , the decoder will Ze Un 、Zs Un , Zc Un 、F Un Fusion processing into New_Mel Un , the sentiment classifier will be Ze Un Perform sentiment classification to get Result Un ; In S304, the emotion encoder is from St_Mel_Aug Un Extract sentiment embedding vector Ze_Aug Un , speaker encoder from Mel-Aug Un Extract speaker embedding vector Zs_Aug Un , content encoder from Mel-Aug Un Extract content embedding vector Zc_Aug Un , the decoder is used to convert Ze_Aug Un 、Zs_Aug Un 、Zc_Aug Un 、F Un Fusion processing into New_Mel_Aug Un .

6. The method for converting emotional speech from source emotion to target emotion according to claim 5, characterized in that: Loss n The expression is: In the formula, L_C n represents sentiment classification loss, C represents sentiment label, and P(.) represents probability function; L_MI n represents the mutual information loss, Indicates the vCLUB method; L_rec n represents the first reconstruction loss; ||.||1 represents the L1 loss function; L_siamese n represents the secondary reconstruction loss.

7. The method for converting emotional speech from source emotion to target emotion according to claim 1, characterized in that: The emotion encoder comprises: a two-dimensional convolution part, a recurrent neural part, a fully connected part, and an activation part connected in sequence; The two-dimensional convolution part includes: 6 two-dimensional convolution blocks connected in sequence; any two-dimensional convolution block includes: 1 2D convolution layer, 1 BatchNorm layer, and 1 Relu activation function connected in sequence; The recurrent neural part includes: a dimension adjustment layer, a GRU network layer, and a dimension squeezing layer connected in sequence; The fully connected part includes two FC layers connected in sequence; The activation part is a Relu activation function.

8. The method for converting emotional speech from source emotion to target emotion according to claim 1, characterized in that: The sentiment classifier includes: three FC layers connected in sequence.

9. An emotional speech conversion device from a source emotion to a target emotion, characterized in that: It uses the emotional speech conversion method from source emotion to target emotion as described in any one of claims 1-8; The emotional speech conversion device from the source emotion to the target emotion comprises: A voice acquisition module, which is used to acquire voice data Voice1 with source emotion and voice data Voice2 with target emotion; The feature processing module is used to extract the features of Voice1 to obtain the fundamental frequency information F1 and the original Mel spectrum Mel1; it is also used to extract the features of Voice2 to obtain the original Mel spectrum Mel2; it performs the first-order time domain difference calculation and the second-order time domain difference calculation on Mel2 in turn to obtain the first-order difference map Δf2 and the second-order difference map Δ 2 f2; Mel2, Δf2, Δ 2 f2 is spliced ​​to obtain the Mel spectrum splicing image St_Mel2; The model processing module is used to input F1, Mel1, and St_Mel2 into the trained emotional speech conversion model for processing to obtain the reconstructed Mel spectrogram New_Mel3; as well as The voice code restoration module is used to perform voice code restoration on New_Mel3 to obtain the speech data Voice3 with the target emotion.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for emotional speech conversion from source emotion to target emotion as described in any one of claims 1 to 8 are implemented.

Images