Text-free speech emotion conversion using discrete and decomposed representations

Abstract

In one embodiment, a method includes accessing a speech signal corresponding to a source emotion; generating a content unit based on the voice signal; generating a changed content unit of the content unit based on the target emotion; determining a corresponding duration of each of the changed content units based on the target emotion; generating a respective tone curve for each of the changed content units based on the target emotion and the respective changed duration; and generating a changed speech signal corresponding to the target emotion based on the target emotion, a speech characteristic associated with the speaker, each changed content unit based on its respective changed duration, and a tone profile of each changed content unit.

Description

Technical Field

[0001] The present disclosure relates generally to speech processing and, in particular, to hardware and software for speech processing. Background Art

[0002] Speech processing is the study of speech signals and signal processing methods. Signals are usually processed in digital form, so speech processing can be seen as a special case of digital signal processing applied to speech signals. Aspects of speech processing include the acquisition, manipulation, storage, transmission and output of speech signals.

[0003] Speech translation is the process of instantly translating a phrase spoken in a conversation into a second language and speaking it out loud in the second language. This is different from phrase translation, where the system only translates a fixed and limited set of phrases that have been manually input into the system. Speech translation technology enables speakers of different languages ​​to communicate. Therefore, speech translation technology is of great value to humanity in science, cross-cultural communication, and global business. Summary of the invention

[0004] According to one aspect, a method is provided, the method comprising: by one or more computing systems: accessing a speech signal corresponding to a source emotion; generating a plurality of content units based on the speech signal; generating a plurality of changed content units of the plurality of content units based on a target emotion; determining a corresponding duration of each of the plurality of changed content units based on the target emotion; generating a corresponding pitch curve of each of the plurality of changed content units based on the target emotion and the corresponding changed duration; and generating a changed speech signal corresponding to the target emotion based on the target emotion, speech characteristics associated with a speaker, a plurality of changed content units based on their corresponding changed durations, and a plurality of pitch curves of the plurality of changed content units.

[0005] Generating the plurality of altered content units may include translating non-verbal vocalizations associated with the speech signal while preserving lexical content associated with the speech signal.

[0006] The source emotion or target emotion may be based on one or more of: prosodic features; speaking style; or non-verbal vocalizations.

[0007] The speech signal may be based on an audio waveform. Generating the plurality of content units may include applying an encoder to the audio waveform.

[0008] The encoder may output a continuous spectral representation of the speech signal.The method may further comprise applying a clustering algorithm to the continuous spectral representation based on a vocabulary associated with the speech signal.

[0009] Generating the plurality of changed content units may include one or more of: changing a content unit in the plurality of content units; adding a content unit to the plurality of content units; or deleting a content unit from the plurality of content units.

[0010] The speech signal may be associated with a speaker. The method may further include generating speech characteristics of the speaker based on the speech signal.

[0011] Generating the plurality of changed content units may be based on a sequence-to-sequence model.

[0012] The sequence-to-sequence model may include: an encoder shared among a plurality of source emotions including the source emotion; and a decoder shared among a plurality of target emotions including the target emotion.

[0013] The sequence-to-sequence model may include: a plurality of encoders respectively dedicated to a plurality of source emotions including the source emotions. The sequence-to-sequence model may include: a plurality of decoders respectively dedicated to a plurality of target emotions including the target emotions.

[0014] The sequence-to-sequence model may include: one encoder shared among a plurality of source emotions including the source emotion. The sequence-to-sequence model may include: a plurality of decoders respectively dedicated to a plurality of target emotions including the target emotion.

[0015] According to another aspect, one or more computer-readable non-transitory storage media are provided, the one or more computer-readable non-transitory storage media containing software that, when executed, is operable to perform the method set forth above.

[0016] According to another aspect, a system is provided, which includes: one or more processors; and a non-volatile memory coupled to the one or more processors, the non-volatile memory including instructions executable by the one or more processors, and the one or more processors being operable to perform the method stated above when executing the instructions.

[0017] In a particular embodiment, a speech processing system may use a model for changing the emotion in a speech signal. In the case of emotion in speech, the error rate of automatic speech recognition (ASR) is usually high. Therefore, by removing emotion from the speech signal, the model can improve automatic speech recognition. On the other hand, the model can also add the desired emotion to the speech signal to generate an emotionally expressive speech signal for training a speech recognition model that can process speech signals with emotion. Where to add the desired emotion (e.g., yawn) can be learned based on a probabilistic model. The model can regard changing emotions as a machine translation task, where the input is a speech utterance with a source emotion and the output is the same utterance with a target emotion. The model can decompose the speech signal into discrete learning representations, which include phonetic content units, prosodic features, speakers, and emotions. Then, the model can modify the speech content by translating the phonetic content units into target emotions and predict prosodic features based on these units. Finally, a speech waveform of the target emotion can be generated by applying a neural vocoder to the predicted representation. Although this disclosure describes particular speech processing performed in a particular manner, this disclosure contemplates any suitable speech processing performed in any suitable manner.

[0018] In a particular embodiment, the speech processing system may access a speech signal corresponding to a source emotion. The speech processing system may then generate a plurality of content units based on the speech signal. In a particular embodiment, the speech processing system may generate a plurality of changed content units of the plurality of content units based on a target emotion. The speech processing system may then determine a corresponding duration of each of the plurality of changed content units based on the target emotion. The speech processing system may then generate a corresponding pitch curve of each of the plurality of changed content units based on the target emotion and the corresponding changed duration. In a particular embodiment, the speech processing system may also generate a changed speech signal corresponding to the target emotion based on the target emotion, speech characteristics associated with the speaker, a plurality of changed content units based on their corresponding changed durations, and a plurality of pitch curves of the plurality of changed content units.

[0019] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. A specific embodiment may include all, some, or none of the components, elements, features, functions, operations, or steps in the embodiments disclosed herein. Embodiments according to the present invention are specifically disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentionally backreferenced any previous claim (especially multiple dependencies) may also be claimed, so that any combination of multiple claims and their multiple features is disclosed and can be claimed regardless of the dependencies selected in the attached claims. The subject matter that can be claimed includes not only multiple combinations of multiple features stated in the attached claims, but also any other combination of multiple features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any embodiment and any feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or any feature described or depicted herein or in any combination with any feature in the appended claims.

[0020] It will be understood that any feature described herein as being suitable for incorporation into one or more aspects or embodiments of the present disclosure is intended to be universal in any and all aspects and embodiments of the present disclosure. Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. The foregoing general description and the following detailed description are exemplary and explanatory only and do not limit the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An example pipeline for speech processing is shown.

[0022] Figure 2 shows the sequence-to-sequence sentiment translation component E s2s Example architecture for .

[0023] Figure 3A and Figure 3B Example mean opinion scores (MOS) and sentiment mean opinion classification (eMOC) scores of our approach are shown, along with an evaluation baseline.

[0024] Figure 4An example confusion matrix for ground-truth records is shown.

[0025] Figure 5 An example confusion matrix of our approach is shown.

[0026] Figure 6 An example confusion matrix for Sequence-to-Sequence - Emotional Speech Conversion (Seq2Seq-EVC) is shown.

[0027] Figure 7 An example confusion matrix for Tacotron2 is shown.

[0028] Figure 8 An example confusion matrix for VAW-GAN is shown.

[0029] Fig. 9 An example method for changing emotion in a speech signal is shown.

[0030] Fig.10 An example computer system is shown. DETAILED DESCRIPTION

[0031] In a particular embodiment, a speech processing system may use a model for changing the emotion in a speech signal. In the case of emotion in speech, the error rate of automatic speech recognition (ASR) is usually high. Therefore, by removing emotion from the speech signal, the model can improve automatic speech recognition. On the other hand, the model can also add the desired emotion to the speech signal to generate an emotionally expressive speech signal for training a speech recognition model that can process speech signals with emotion. Where to add the desired emotion (e.g., yawn) can be learned based on a probabilistic model. The model can regard changing emotions as a machine translation task, where the input is a speech utterance with a source emotion, and the output is the same utterance with a target emotion. The model can decompose the speech signal into discrete learning representations, which include phonetic content units, prosodic features, speakers, and emotions. Then, the model can modify the speech content by translating the phonetic content units into target emotions, and predict prosodic features based on these units. Finally, a speech waveform of the target emotion can be generated by applying a neural vocoder to the predicted representation. Although this disclosure describes particular speech processing performed in a particular manner, this disclosure contemplates any suitable speech processing performed in any suitable manner.

[0032] In a particular embodiment, the speech processing system may access a speech signal corresponding to a source emotion. The speech processing system may then generate a plurality of content units based on the speech signal. In a particular embodiment, the speech processing system may generate a plurality of changed content units of the plurality of content units based on a target emotion. The speech processing system may then determine a corresponding duration of each of the plurality of changed content units based on the target emotion. The speech processing system may then generate a corresponding pitch curve of each of the plurality of changed content units based on the target emotion and the corresponding changed duration. In a particular embodiment, the speech processing system may also generate a changed speech signal corresponding to the target emotion based on the target emotion, speech characteristics associated with the speaker, a plurality of changed content units based on their corresponding changed durations, and a plurality of pitch curves of the plurality of changed content units.

[0033] Speech emotion conversion is the task of modifying the perceived emotion in a speech utterance while preserving the lexical content and the speaker's identity. The embodiments disclosed herein convert the emotion conversion problem into a spoken translation task. We can use a decomposition that makes the speech signal into discrete learning representations, which include phonetic content units, prosodic features, speakers, and emotions. First, we can modify the speech content by translating the phonetic content units into target emotions, and then predict the prosodic features based on these units. Finally, the speech waveform can be generated by feeding the predicted representation to a neural vocoder. This paradigm can allow us to go beyond the spectral and parameter changes of the signal and model non-linguistic vocalizations (such as laughter insertion, yawn removal, etc.). We objectively and subjectively show that our method disclosed herein is superior to traditional methods in terms of perceived emotion and audio quality, and even outperforms text-based systems. We rigorously evaluate all components of this complex system and infer with extensive model analysis and ablation studies to better emphasize the architectural choices, advantages, and other aspects of our method.

[0034] Generating natural-sounding spoken utterances and dialogues may be a requirement for improving human-computer interaction. One of the main obstacles to improving the naturalness of speech generation may be modeling emotional states that are emotionally expressive. The difficulty may be that emotion is a phenomenon that affects all linguistic levels simultaneously: when a person goes from a happy state to an angry state, the person may use different vocabulary, insert non-verbal vocalizations (crying, grunting, etc.), modify the prosody (intonation and rhythm), and change the quality of the voice due to stress. Vice versa, each of the levels may contribute to the perception of the speaker's emotional state, where non-verbal aspects may often be more important than lexical content.

[0035] Existing emotion generation techniques or emotion conversion techniques may struggle to produce convincing results because they may only manage to process a subset of these levels. In short, signal-based methods may focus primarily on manipulating the parameters of the speech signal and may only process variations in the level of voice and prosody. In contrast, text-based methods can generate emotionally expressive speech but have difficulty with non-verbal vocalizations because these are usually not annotated in speech corpora.

[0036] The embodiments disclosed herein focus on the task of speech emotion conversion in a parallel dataset setting to modify the perceived emotion in speech utterances while preserving speaker identity and lexical content.

[0037] A pipeline for speech processing based on the embodiments disclosed herein may include four main blocks: a speech tokenizer, a content translation model, a prosody prediction model, and a neural vocoder. We can start by extracting a discrete representation of a speech signal. In a particular embodiment, the speech signal may be associated with a speaker. The speech processing system may generate speech characteristics of the speaker based on the speech signal. We can translate the representation into a target emotion while retaining the lexical content (e.g., removing laughter content, inserting yawn content). We can then predict prosodic features based on the translated representation. The neural vocoder may synthesize a speech waveform based on the translated phonetic content, the predicted prosody, the speaker label, and the target emotion label. Figure 1 An example pipeline 100 for speech processing is shown. The content encoder E c 110 encodes the input signal 105 into a discrete sequence of content units 115. Next, a sequence to sequence (S2S) model 120 may be applied to translate between multiple sequences corresponding to different emotions. The S2S model may be based on the emotion Z emo 135. As an example and not limitation, the input to the S2S model 120 can be a discrete sequence of content units 115 based on the source sentiment, and the output of the S2S model can be a discrete sequence of content units 125 based on the target sentiment. Then, we can use the sentiment Z emo 135 is the conditional duration prediction model E dur 130 to predict duration 140. Then, we can use emo 135 Conditional F0 Estimation Model E F0 145 to predict pitch. Speaker identity Z spk 150, predicted duration 140, predicted pitch and emotion (Zemo )135 can be fed to a vocoder (G)155, which can generate an output waveform 160.

[0038] The contribution of the embodiments disclosed herein may be twofold: We disclose a new text-free approach by converting the speech emotion conversion task into a spoken language translation problem. We show how this paradigm can be used to model emotionally expressive non-verbal communication cues and generate high-quality speech samples. Finally, we show for the first time simultaneous coverage of all levels of emotionally expressive speech modeling.

[0039] Experimental results show that our approach outperforms signal-only emotion conversion techniques and text-based methods in terms of both generated quality and perceived emotion. We conduct an extensive evaluation of the modules that make up our system and perform inference with ablation experiments to better understand the impact of each decomposed component on the overall generated speech.

[0040] Since emotions may manifest themselves in multiple aspects of spoken language, one may need to consider all aspects of the conversion process in order to optimally convert emotions. In certain embodiments, source or target emotions may be based on one or more of: prosodic features; speaking style; or non-verbal vocalizations. As an example and not limitation, emotions may be expressed via prosodic features (high pitch, slow speech rate, etc.), speaking style (shouting, whispering, etc.), and non-verbal vocalizations (laughing, yawning, crying, etc.).

[0041] The embodiments disclosed herein can use the decomposed representation of the speech signal to synthesize speech at the target emotion. We can consider four components in the decomposition: c , (zdur, zF0), zspk, zemo represent the phonetic content, prosodic features (i.e., F0 and duration), speaker identity, and emotion label.

[0042] Specifically, the embodiments disclosed herein may be based on the following cascaded pipeline: (i) extracting z from the original waveform using a self-supervised learning (SSL) model; c ; (ii) translation cNon-verbal vocalizations in, while preserving the lexical content (e.g., when transitioning from being amused to sleepy, we can remove the laughter and insert a yawn); in other words, generating multiple modified content units can include: translating the non-verbal vocalizations associated with the speech signal while preserving the lexical content associated with the speech signal; (iii) predicting the prosodic features of the target emotion based on the translated content; (iv) synthesizing speech according to the translated content, the predicted prosody, the target speaker identity, and the target emotion label.

[0043] To represent the speech phonetic content, we can use a pre-trained SSL model (i.e., HuBERT) to extract the discrete representation of the audio signal. We can use the SSL representation for the phonetic content in order to capture non-verbal vocalizations (unlike text, non-verbal vocalizations are usually not annotated). We can discretize this representation for better modeling and sampling (instead of regressing on continuous variables). This paradigm can allow us to benefit from all the latest advances in natural language processing (NLP).

[0044] by denote the domain of the audio sample. Thus, the representation of the audio waveform can be a sequence of samples x = (x 1 ,..., x T ), where for all 1 < τ < T, each x i ∈ X. The content encoder E c can be the HuBERT model pre-trained on an experimental speech corpus. Similar to BERT, HuBERT is a self-supervised model with the training task of masked prediction on continuous audio signals. During training, the targets can be obtained by clustering the Mel Frequency Cepstral Coefficients (MFCCs) features or the learned representations from earlier iterations. In a particular embodiment, the speech signal can be based on the audio waveform. Accordingly, generating multiple content units can include applying the encoder to the audio waveform. The input to the content encoder E c can be the audio waveform x, and the output can be a spectral representation sampled at a lower frequency where L < T. In other words, this encoder can output a continuous spectral representation of the speech signal. In a particular embodiment, the speech processing system can apply a clustering algorithm to the continuous spectral representation based on the vocabulary associated with the speech signal. Since HuBERT outputs continuous representations, an additional k-means step may be required in order to quantize these representations into those by (where and K can be a sequence of discrete units represented by the vocabulary). In the rest of this disclosure, we refer to these discrete representations as "units". We can extract representations from layer 9 of the HuBERT model and set k = 200. In a specific embodiment, repeated units can be omitted (e.g., 0, 0, 0, 1, 1, 2 - + 0, 1, 2). We use "deduped" to represent such a sequence. We can use HuBERT for phonetic content units because it can better disentangle the speech content from both the speaker and the prosody than other SSL-based models.

[0045] The embodiments disclosed herein can convert speech emotions while keeping the speaker identity fixed. To this end, we can construct a speaker representation z spk , and included it as an additional regulator during the waveform synthesis stage. spk , we can optimize the parameters of a fixed-size lookup table. Although this modeling may limit our ability to generalize to new and unseen speakers, it can produce higher quality generations. Alternatively, we try to set z spk is represented as a d-dimensional vector. However, we observe that this approach preserves the prosodic characteristics of the source emotion, making disentanglement poorer during the waveform synthesis stage.

[0046] We can represent the sentiment label using a categorical variable represented by a 1-hot vector. We observed that this component controls the timbre characteristics (e.g., roughness, smoothness, etc.) of the generated speech signal during the synthesis stage.

[0047] The embodiments disclosed herein can use the aforementioned representations to synthesize speech signals at the target emotion. We can use a translation model to convert between phonetic content units of the source emotion to phonetic content units of the target emotion. This can serve as a learnable insertion / deletion / replacement mechanism for non-linguistic utterances while preserving lexical content (e.g., removing yawns while preserving linguistic content). Next, we can predict prosodic features (duration and F0) based on the translated phonetic content units and the target emotion label, and inflate the sequence according to the predicted duration. This can be used later as a conditioning effect on the waveform synthesis stage.

[0048] To translate the speech content units, we can use the s2s Represents a sequence-to-sequence transformer model. Figure 2 shows the sequence-to-sequence sentiment translation component E s2sExample architecture 200 of 210. In a particular embodiment, generating multiple changed content units can be based on a sequence-to-sequence model. Here, different encoders and decoders can be used for each emotion, but we have also tested a shared architecture, which will be disclosed later in this disclosure. In one embodiment, the sequence-to-sequence model may include: an encoder shared among multiple source emotions including the source emotion; and a decoder shared among multiple target emotions including the target emotion. In another embodiment, the sequence-to-sequence model may include multiple encoders that are respectively dedicated to multiple source emotions including the source emotion. The sequence-to-sequence model may include multiple decoders that are respectively dedicated to multiple target emotions including the target emotion. In yet another embodiment, the sequence-to-sequence model may include an encoder shared among multiple source emotions including the source emotion. The sequence-to-sequence model may include multiple decoders that are respectively dedicated to multiple target emotions including the target emotion. To E s2s The input of 210 may be a source emotion The model can be trained to output the phonetic content units. The sequence 210 includes the same lexical content with additions / deletions / replacements of speech cues associated with emotional expressions (e.g., insertion of a laughter unit). In other words, generating the plurality of changed content units may include one or more of: changing a content unit in the plurality of content units; adding a content unit to the plurality of content units; or deleting a content unit from the plurality of content units. The optimization may be performed by optimizing the predicted unit at each position in the sequence. The cross-entropy (CE) loss between the ground truth units is minimized as,

[0049]

[0050] In our experiments, we observed that directly optimizing the above model for translation can capture emotional transfer, but may not maintain the same lexical content, resulting in emotionally expressive but unintelligible speech utterances (e.g., the model can add laughter, but will corrupt the sentence by removing the desired syllables). To reduce this problem, similar to BART, we can pre-train the translation model on the task of language denoising autoencoders. To better support translation between all emotions, we can use dedicated encoders and decoders for each emotion (see Figure 2 ). In addition, we evaluate our models with a shared encoder and a shared decoder conditioned on the target sentiment, as well as a model with only a shared encoder. We evaluate all methods and summarize the results.

[0051] Next, we can use the translated sequence of phonetic content units to predict the prosodic features corresponding to the target emotion. We can view the prosodic representation as a tuple of content unit duration and F0.

[0052] We can start by describing the duration prediction process. Since we are processing deduplicated sequences, we may first need to predict the duration of each phonetic content unit. We can use a convolutional neural network (CNN) to learn the mapping between content units and durations. We use E dur To represent the model. dur During the training, we can input the phonetic content unit z with duplicate data removed c , and use the ground truth phonetic content unit duration as supervision. We minimize the mean squared error (MSE) between the network's output and the target duration. We also evaluate duration prediction models based on n-grams. The n-gram model is trained by computing the mean frequency μ and standard deviation σ of each n-gram in the training set. During inference, we predict the duration of each n-gram by sampling from N(μ,σ). For unseen n-grams, we fall back to a smaller n-gram model.

[0053] We now turn to describing the F0 prediction process. We can use the F0 estimation model to estimate the F0 content of a unit z according to its phonetic content. c We use E F0 The model represented can be CNN, and then A linear layer that projects the output. The final activation layer can be set to sigmoid so that the network outputs [0,1] d During training, we can use the YAAPT algorithm to extract F0 as a target. Next, we can normalize the F0 values ​​using the mean and standard deviation for each speaker. We can discretize the range of F0 values ​​into d bins represented by one-hot encodings. Next, we can apply Gaussian blur to these encodings to obtain The final supervision goal is represented by and d = 50. Formally, we can minimize the binary cross-entropy (BCE) between each coordinate of the target and the network output as

[0054]

[0055] During inference, multiple frequency bins can be activated to different degrees. We can output an F0 value corresponding to a weighted average of the activated bins. When converting bins back to F0 values, this modeling can allow for a better output range, rather than a single representative F0 value per bin. For the conversion of F0 to bins, we can use an adaptive bin strategy so that the probability mass of each bin is the same. For completeness, we explored uniform bins and summarize the results later in this disclosure. Additional log-F0 estimation results will be presented later in this disclosure.

[0056] Note that the mapping between discrete unit sequences and prosodic features (F0 and duration) can be one-to-many, since this may depend on the target emotion. Therefore, we can additionally make E dur and E F0 These two are due to The two models can be trained independently.

[0057] We can use a variation of the High Fidelity Generative Adversarial Network (HiFi-GAN) neural vocoder. The architecture of the HiFi-GAN consists of a generator G and a set of discriminators D. We can adapt the generator component to take as input a sequence of inflated phonetic content units predicted using: predicted duration; predicted F0; target speaker embedding; and target emotion label. The above features can be concatenated along the time axis and fed into a sequence of convolutional layers that output a 1D signal. The sampling rates of the unit sequence and F0 can be matched by linear interpolation, while the speaker embedding and emotion label can be copied.

[0058] The discriminator can include two groups: multi-scale discriminator (MSD) and multi-period discriminator (MPD). The first type can operate on sliding windows of different sizes on the signal (2, 4), while the latter can sample the signal at different rates (2, 3, 5, 7, 11). In general, each discriminator D can be trained by minimizing the following loss function i

[0059]

[0060] in, is the time domain signal reconstructed from the decomposed representation. There can be two additional loss terms for optimizing G. The first can be the mean-absolute-error (MAE) reconstruction loss in the log-mel frequency domain in is a spectral operator that computes a Mel-spectrogram. The second loss term can be a feature matching loss that penalizes large differences in the intermediate discriminator representations. where ξ j is to extract the discriminator D with R layers i The overall goal of optimizing the system can be:

[0061]

[0062] Among them, λ fm =2 and λ r =45.

[0063] The embodiments disclosed herein use the Experimental Emotional Sound Database to train and evaluate our model. The Experimental Emotional Sound Database includes 7,000 speech utterances. Each transcript is recorded by multiple native speakers (two male speakers and two female speakers) with multiple performance emotions (neutral, amused, angry, sleepy, bored). This allows us to create a dataset of utterance pairs for translation tasks. Specifically, we create utterance pairs based on the same transcript but recorded with different performance emotions. Since the size of the dataset is small (about 9 hours), we further expand the dataset by creating additional similar (parallel) pairs from different speakers. Overall, the size of the entire dataset is 78,324 pairs. We split the data into training set / validation set / test set in a ratio of 90 / 5 / 5, so that there is no overlap of utterances between these sets. In our experiments, random splitting (e.g., overlapping transcripts) leads to memorizing utterances and not being able to generalize to unseen data.

[0064] In a particular embodiment, the training data may include speech data having the same content but at different emotions. The speech data may be from different speakers. Each component, i.e., the content encoder 110, the sequence-to-sequence model 120, the duration prediction model 130, the F0 estimation model 145, and the vocoder 155, may be trained separately. Each of the trained components may then be combined to obtain a pipeline 100 for speech processing.

[0065] In certain embodiments, pipeline 100 can have different use cases. In one use case, pipeline 100 can be used to improve automatic speech recognition (ASR) by preprocessing speech to make it neutral (e.g., remove emotion). Alternatively, pipeline 100 can be used to generate emotional training data from neutral training data. The emotional training data can then be used to train an ASR model. Another use case can be to use pipeline 100 to change the emotion of speech. As an example and not limitation, when the speaker can be predetermined or generated, the application can select a target emotion and use pipeline 100 to process the input speech signal.

[0066] We experimented with the sequence-to-sequence transformer model implemented in fairseq. The model contains three layers of both the encoding and decoding modules, four attention heads, an embedding size of 512, an FFN size of 512, and a dropout probability of 0.1. For pre-training, we used a mixture of the experimental speech corpus, experimental audiobook recordings, and the experimental emotional sound database, and stopped after 3M update steps. We used the following input augmentations: infilling with λ=3.5 for the Poisson distribution, token masking with probability 0.3, random masking with probability 0.1, and sentence permutation. Finally, we fine-tuned the model on the translation task using paired utterances of different emotions from the experimental emotional sound database and used early stopping using the loss from Eq. (1). Our F0 prediction model E F0 It consists of six 1D convolutional layers, where the number of kernels in each layer is 256 and the individual kernel size is 5. For nonlinearity, we used the ReLU activation function followed by layer-norm and dropout (p=0.1). Our duration prediction model E dur Two convolutional layers were included, where the number of kernels in each layer was 256 and the size of each kernel was 3. For nonlinearity, we used the ReLU activation function, followed by layer normalization and dropout (p=0.5). All experiments in the embodiments disclosed herein were performed using 8 GPUs, each with 32GB of memory.

[0067] We compare our approach to a text-free speech emotion conversion method, VAW-GAN, and the state-of-the-art (SOTA) text-based emotional voice conversion model, Seq2seq-EVC. We also evaluate an emotionally expressive text-to-speech (TTS) system based on Tacotron2. We consider text-based systems as systems that use text annotations during training.

[0068] For text-based methods, we used the Tacotron2 model and the Seq2seq-EVC model. The input to Tacotron2 is the ground truth text representing the speech content. We modify the Tacotron2 architecture by adding a global style token for controlling the target emotion. The input to Seq2seq-EVC is the ground truth phoneme coupled to the source speech utterance. The output of both systems is a mel-spectrogram of the speech utterance at the target emotion. To reconstruct the time domain signal, we used the HiFi-GAN vocoder. All baselines are trained and evaluated on the Experimental Emotional Voice Database. Seq2seq-EVC and Tacotron2 are first pre-trained on the Experimental Voice Cloning Dataset.

[0069] For speech emotion conversion, the embodiments disclosed herein use a new subjective metric called emotion-mean-opinion-classification (eMOC). In the eMOC study, human raters were presented with speech utterances and a set of emotion categories. The raters were instructed to select the emotion that best fits the speech utterance. As an example and not a limitation, for eMOC, the raters were asked: "Select the emotion that best fits the given speech utterance for a given number of emotion categories." All raters were native English speakers located in the United States. The eMOC score is the percentage of raters who selected the target emotion given a speech recording. The final score is the average of all raters and utterances in the study. In addition, we measured the perceived audio quality using the mean-opinion-score (MOS). As an example and not a limitation, for the MOC metric, the raters were asked: "Rate the quality and naturalness of a given speech utterance on a scale of 1 to 5 (1 is low quality and low naturalness, and 5 is high quality and high naturalness)." All raters were native English speakers located in the United States. For “amused”, “angry”, “bored”, and “sleepy”, we used samples converted from “neutral”. The CrowdMOS package was used in all subjective experiments, which recommends methods for removing outliers. Participants were recruited using a crowd-sourcing platform.

[0070] Embodiments disclosed herein perform speech emotion conversion while preserving the lexical content of the speech signal. However, to compare different translation models with different vocabularies, one cannot simply use metrics such as the Bilingual Evaluation Underscore (BLEU). Therefore, we report the word error rate (WER) metric and the phoneme error rate (PER) metric extracted using a pre-trained SOTA ASR system. We used the BASEwav2vec 2.0 phoneme detection model, which was trained on a 960-hour experimental speech corpus with a chained temporal classification (CTC) loss from scratch. Since ASR models may experience performance degradation when evaluating emotionally expressive speech, the WER metric and the PER metric are reported when the emotion is converted to neutral: {amused, angry, sleepy, bored} - neutral.

[0071] We first use objective metrics to independently tune different components of our system: the content unit extraction configuration and the prosody modeling module (F0 estimator and duration estimator). Next, we perform a subjective evaluation of our best system in terms of audio quality (MOS) and perceived emotion (eMOC) and compare our approach to the baseline. Finally, we perform ablation experiments in which we measure the impact of each component on the perceived emotion.

[0072] The intermediate HuBERT representations obtained from different layers may have an impact on the downstream task at hand. The number of units (k) can also have an impact on the overall performance of the system. To better understand the effect of these architectural configurations in our setting, we experimented with extracting HuBERT features from layers 6 and 9, using 100 and 200 clusters for a k-means post-processing step. Additionally, we measured the impact of pre-training the translation model on performance. When we compare models with different vocabularies, we may not use metrics such as BLEU. Therefore, we report WER and PER. The results are in Table 1.

[0073]

[0074]

[0075] Table 1: Evaluation of different token extraction configurations and the effect of pre-training the translation model. #units denotes the vocabulary size, and the layer number denotes the layer index used in HuBERT.

[0076] For sentiment conversion, we found that using the 9th HuBERT layer and 200 tokens performed best. It can be seen that without pre-training, all models failed to generate understandable speech. This may be due to the limited number of similar sentiment training pairs. Evaluation of different model design architectures will be disclosed later in this disclosure.

[0077] We evaluate the F0 estimation model using the mean absolute error (MAE) between the ground truth F0 and the predicted F0. In this experiment, we explored multiple configurations for training such an estimator. Specifically, we evaluated different interval strategies, normalization methods, and prediction rules. For interval strategies, we explored adaptive intervals versus uniform intervals. Under normalization, we explored no normalization, mean normalization, and mean and standard deviation normalization (the mean and standard deviation were calculated using the F0 values ​​for each speaker). Finally, in addition to the weighted average prediction rule described previously, we also evaluated the argmax prediction rule that predicted the interval with the highest score. The results are summarized in Table 2. The results show that the weighted average prediction rule performs better than argmax, especially when combined with adaptive intervals. This can be explained by the large range of intervals in the adaptive case, which results in a larger MAE when selecting a single interval using the argmax operator. Although adaptive quantization achieves the best performance, uniform quantization can achieve comparable results in specific settings. For normalization, it may be better to perform normalization, as the specific normalization method may have a small impact on the performance.

[0078]

[0079] Table 2: Evaluation of different F0 estimation configurations. MAE is reported only for voiced frames.

[0080] Next, we evaluate the duration prediction models using the MAE between the target duration and the predicted duration. For a more complete analysis, we also report the accuracy using thresholds of 0 milliseconds (ms), 20ms, and 40ms. We explored the CNN duration predictor and three n-gram-based models. The results are summarized in Table 3. As expected, the CNN outperforms the n-gram model, with an accuracy of about 94% when considering a tolerance level of 40ms.

[0081]

[0082]

[0083] Table 3: Evaluation of four duration prediction models

[0084] Figure 3A and Figure 3B The MOS and eMOC scores of our method and the evaluation baselines are shown. Figure 3A Mean opinion scores (MOS) measuring perceived audio quality are reported. We report the mean score with a 95% confidence interval. Figure 3BEmotion Mean Opinion Classification (eMOC) scores, which measure the perceived emotion, are reported. We report the average score for each emotion (chance level: 25%). The results show that our method outperforms the baselines in both MOS and eMOC. Although Tacotron2 and Seq2seq-EVC may successfully convey the target emotion, they may produce less natural and emotionally expressive speech utterances, which is reflected in the lower MOS. Unlike our method, both Tacotron2 and Seq2seq-EVC are text-based systems. Therefore, they may try to learn the alignment (attention map) between the text input and the audio target. This task may be particularly challenging because non-verbal cues (e.g., laughter, breathing) may not be annotated in the text input. We can assume that this misalignment may lead to less natural and emotionally expressive speech generation. We additionally provide the eMOC confusion matrix, evaluation baselines, and ground truth records of the proposed method. Figure 4 An example confusion matrix for the ground truth records is shown. Figure 5 An example confusion matrix of our approach is shown. Figure 6 An example confusion matrix for Seq2Seq-EVC is shown. Figure 7 An example confusion matrix for Tacotron2 is shown. Figure 8 An example confusion matrix for VAW-GAN is shown.

[0085] Recall that the embodiments disclosed herein can use a decomposed speech representation comprising four feature groups. In a particular embodiment, we can measure the effect of each feature by gradually adding different components and evaluating their impact on the eMOC metric. Specifically, we can start by evaluating the source features and replacing the sentiment tokens. Then, we can predict the unit duration and F0 of the target sentiment. Finally, the full effect of our method can be achieved by combining the unit translation model. For reference, we report the results of the original recording, the results of the resynthesized recording (i.e., using only the source features), and the results of the target recording. The results are summarized in Table 4.

[0086] The results show that our approach is comparable to the ground truth recordings in terms of perceived emotions. Interestingly, for the emotion “sleepy”, modifying the timbre and adjusting the unit duration using the target emotion token was enough to achieve 70.31%, while the remaining emotions required further processing. For “amused”, “angry” and “bored”, modifying F0 achieved a performance 5% lower than the ground truth recordings. When the entire pipeline is applied, these results are comparable to the ground truth. This can be explained by the addition and removal of non-verbal utterances by the translation model.

[0087]

[0088] Table 4: The impact of the components in our system on the perceived emotion. “original-neutral” represents the ground truth neutral transcript, while “original-emotional” represents the ground truth emotional transcript. “src”, “tgt”, and “pred” represent the features extracted from the source speech, the features extracted from the target speech, and the features predicted by our system, respectively.

[0089] Due to the size of the Experimental Speech Corpus (7000 samples in total), the number of unique utterances is small. Therefore, the model can memorize utterances and may not generalize to unseen sentences. Therefore, we tried to convert out-of-domain recordings. To do this, we fed our system with recordings from the Experimental Speech Corpus. Since the Experimental Speech Corpus includes samples with no emotional expression, we regard it as "unemotional". We converted these samples to different emotions and evaluated the performance. For evaluation, we randomly sampled 20 utterances converted to various emotions. Our method achieved an average eMOC score of 82.25% ± 7.32 across all four emotions (amused, angry, bored, sleepy) and an average MOS score of 3.69 ± 0.26. When considering the lexical content, the WER between the ground truth text and the ASR-based transcript of the generated audio is 27.92. These results are similar to those reported on the Experimental Speech Corpus.

[0090] In a specific embodiment, we evaluate three weight sharing schemes for our model: (i) All emotions use the same encoder and decoder components of the conversion architecture. In this case, we make the model conditioned on the target emotion. This can be done by prepending special target emotion tokens at the beginning of the decoding process. We denote this approach as “share-all”. (ii) All emotions share the same encoder, but have separate dedicated decoders. In this case, target emotion conditioning is not required. We denote this approach as “share-enc”. Finally, (iii) Each emotion has a dedicated encoder and decoder. We denote this approach as “share-none”. We evaluate all three approaches and summarize the results in Table 5.

[0091]

[0092] Table 5: Evaluation of three weight sharing schemes for the translation model.

[0093] As can be seen, the shared encoder architecture is comparable to the unshared architecture, while the shared configuration is inferior. Although the shared encoder configuration and the unshared configuration are similar in terms of lexical reconstruction, the unshared configuration generates more emotionally expressive speech in our listening tests.

[0094] We provide the results of the F0 estimation module in Table 6. “Log” means that a logarithmic function is applied before normalization.

[0095]

[0096]

[0097] Table 6: Evaluation of different F0 estimation configurations. MAE is reported only for voiced frames.

[0098] Fig. 9 An example method 900 for changing an emotion in a speech signal is shown. The method may begin at step 910, where a speech processing system may access a speech signal corresponding to a source emotion. At step 920, the speech processing system may generate a plurality of content units based on the speech signal. At step 930, the speech processing system may generate a plurality of changed content units of the plurality of content units based on a target emotion. At step 940, the speech processing system may determine a corresponding duration of each of the plurality of changed content units based on the target emotion. At step 950, the speech processing system may generate a corresponding pitch curve for each of the plurality of changed content units based on the target emotion and the corresponding changed duration. At step 960, the speech processing system may generate a changed speech signal corresponding to the target emotion based on the target emotion, speech characteristics associated with a speaker, a plurality of changed content units based on their corresponding changed durations, and a plurality of pitch curves of the plurality of changed content units. Where appropriate, particular embodiments may be repeated. Fig. 9 One or more steps of the method. Fig. 9 The specific steps of the method are described and shown as being performed in a specific order, but the present disclosure contemplates Fig. 9 In addition, although the present disclosure describes and illustrates a method for changing the emotion in a speech signal, including Fig. 9 However, the present disclosure contemplates any suitable method for changing the emotion in a speech signal, including any suitable steps, which may include, where appropriate, Fig. 9 All, some, or none of the steps in the method Fig. 9 Furthermore, although the present disclosure describes and illustrates the steps of performing Fig. 9 The present disclosure contemplates specific components, devices or systems for performing the specific steps of the method. Fig. 9 Any suitable combination of any suitable components, devices, or systems for any suitable steps of the method.

[0099] Fig.10 An example computer system 1000 is shown. In particular embodiments, one or more computer systems 1000 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1000 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1000 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1000. Herein, references to a computer system may include a computing device, and vice versa, where appropriate. Furthermore, references to a computer system may include one or more computer systems, where appropriate.

[0100] The present disclosure contemplates any suitable number of computer systems 1000. The present disclosure contemplates computer systems 1000 in any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive self-service terminal (kiosk), a mainframe, a network of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these systems. Where appropriate, computer system 1000 may include one or more computer systems 1000; may be single or distributed; across multiple locations; across multiple machines; across multiple data centers; or located in a cloud (which may include one or more cloud components in one or more networks). Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch processing mode. Where appropriate, one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0101] In a particular embodiment, computer system 1000 includes a processor 1002, memory 1004, storage 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having particular numbers of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable arrangement.

[0102] In a particular embodiment, the processor 1002 includes hardware for executing a plurality of instructions, such as those that constitute a computer program. By way of example and not limitation, to execute a plurality of instructions, the processor 1002 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1004, or storage 1006; decode and execute the instructions; and then write one or more results to an internal register, an internal cache, memory 1004, or storage 1006. In a particular embodiment, the processor 1002 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more page table caches (translation lookaside buffers, TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 1004 or storage 1006, and the instruction cache may speed up the retrieval of the instructions by the processor 1002. The data in the data cache may be a copy of the data in the memory 1004 or storage 1006 for the instruction operation executed at the processor 1002; may be the result of the previous instruction executed at the processor 1002, for access by the subsequent instruction executed at the processor 1002, or for writing to the memory 1004 or storage 1006; or may be other suitable data. The data cache may accelerate the read operation or write operation of the processor 1002. The TLB may accelerate the virtual address translation of the processor 1002. In a particular embodiment, the processor 1002 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1002 including any suitable number of any suitable internal registers. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 1002. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0103] In a particular embodiment, the memory 1004 includes a main memory for storing instructions for execution by the processor 1002 or data for operation by the processor 1002. As an example and not limitation, the computer system 1000 can load instructions from the memory 1006 or another source (e.g., another computer system 1000) to the memory 1004. The processor 1002 can then load these instructions from the memory 1004 to an internal register or an internal cache memory. In order to execute these instructions, the processor 1002 can retrieve these instructions from the internal register or the internal cache memory and decode them. During or after executing these instructions, the processor 1002 can write one or more results (which can be intermediate results or final results) to the internal register or the internal cache memory. The processor 1002 can then write one or more of these results to the memory 1004. In a particular embodiment, the processor 1002 executes only instructions in one or more internal registers or in one or more internal cache memories, or in memory 1004 (different from memory 1006 or other locations), and operates only on data in one or more internal registers or in internal cache memories, or in memory 1004 (different from memory 1006 or other locations). One or more memory buses (each memory bus may include an address bus and a data bus) can couple the processor 1002 to the memory 1004. As described below, the bus 1012 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 1002 and the memory 1004 and facilitate access to the memory 1004 requested by the processor 1002. In a particular embodiment, the memory 1004 includes random access memory (RAM). Where appropriate, the RAM is a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1004 may include one or more memories 1004. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0104] In a particular embodiment, the memory 1006 includes a large capacity memory for data or instructions. As an example and not limitation, the memory 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these memories. Where appropriate, the memory 1006 may include a removable medium or a non-removable (or fixed) medium. Where appropriate, the memory 1006 may be inside or outside the computer system 1000. In a particular embodiment, the memory 1006 is a non-volatile solid-state memory. In a particular embodiment, the memory 1006 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage 1006 taking any suitable physical form. Where appropriate, storage 1006 may include one or more storage control units that facilitate communication between processor 1002 and storage 1006. Where appropriate, storage 1006 may include one or more memories 1006. Although the present disclosure describes and illustrates particular storage, the present disclosure contemplates any suitable storage.

[0105] In a particular embodiment, the I / O interface 1008 includes such hardware, software, or both: the hardware, software, or both provide one or more interfaces for communication between the computer system 1000 and one or more I / O devices. Where appropriate, the computer system 1000 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 1000. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a tracking ball, a camera, another suitable I / O device, or a combination of two or more of these I / O devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1008 for the I / O device. Where appropriate, the I / O interface 1008 may include one or more devices or software drivers that enable the processor 1002 to drive one or more of these I / O devices. Where appropriate, I / O interface 1008 may include one or more I / O interfaces 1008. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0106] In a particular embodiment, the communication interface 1010 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1000 and one or more other computer systems 1000 or one or more networks. By way of example and not limitation, the communication interface 1010 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other line-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1010 for the network. By way of example and not limitation, the computer system 1000 may communicate with a network such as an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these networks. One or more portions of one or more of these networks may be wired or wireless. As an example, the computer system 1000 can communicate with the following networks: a wireless PAN (WPAN) (e.g., a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 1000 may include any suitable communication interface 1010 for any of these networks. Where appropriate, the communication interface 1010 may include one or more communication interfaces 1010. Although the present disclosure describes and illustrates particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0107] In particular embodiments, bus 1012 includes hardware, software, or both that couples the various components of computer system 1000 to each other. By way of example and not limitation, bus 1012 may include: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, front-side bus (FSB), Hypertransport (HT) interconnect, Industry Standard Architecture (ISA) bus, InfiniBAND interconnect, low-pin-count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCIe) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these buses. Where appropriate, bus 1012 may include one or more buses 1012. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0108] Thus, a method has been described, the method comprising: accessing a speech signal corresponding to a source emotion; generating a content unit based on the speech signal; generating a changed content unit of the content unit based on a target emotion; determining a corresponding duration of each of the changed content units based on the target emotion; generating a corresponding pitch curve of each of the changed content units based on the target emotion and the corresponding changed duration; and generating a changed speech signal corresponding to the target emotion based on the target emotion, speech characteristics associated with a speaker, each changed content unit based on its corresponding changed duration, and each pitch curve of each changed content unit.

[0109] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives, tapes, solid-state drives (SSDs), RAM drives, secure digital cards (SECURE DIGITAL cards) or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these storage media. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0110] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, "and" is joint and several, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0111] The scope of the present disclosure includes all changes, substitutions, changes, transformations and modifications to the example embodiments described or shown herein, which will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific parts, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the parts, elements, features, functions, operations, or steps described or shown anywhere herein as will be understood by those of ordinary skill in the art. In addition, references to devices or systems that are suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated to perform a specific function, or components in a device or system in the appended claims include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so suitable for, arranged to, capable of, configured to, enabled to, can be operated, or can be operated. Additionally, although this disclosure may describe or illustrate a particular embodiment as providing particular advantages, that particular embodiment may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A method comprising: One or more computing systems: accessing a speech signal corresponding to a source emotion; generating a plurality of content units based on the speech signal; generating a plurality of changed content units of the plurality of content units based on the target sentiment; determining a respective duration of each of the plurality of changed content units based on the target sentiment; generating a corresponding pitch curve for each of the plurality of changed content units based on the target emotion and the corresponding changed duration; A changed speech signal corresponding to the target emotion is generated based on the target emotion, speech characteristics associated with a speaker, the multiple changed content units based on their corresponding changed durations, and the multiple pitch curves of the multiple changed content units.

2. The method according to claim 1, wherein: Generating the plurality of altered content units includes translating non-verbal vocalizations associated with the speech signal while preserving lexical content associated with the speech signal.

3. The method according to claim 1 or 2, wherein: The source emotion or the target emotion is based on one or more of: prosodic features; speaking style; or non-verbal vocalizations.

4. A method according to any preceding claim, wherein: The speech signal is based on an audio waveform, and wherein generating the plurality of content units comprises applying an encoder to the audio waveform.

5. The method according to claim 4, wherein: The encoder outputs a continuous frequency spectrum representation of the speech signal, wherein the method further comprises: A clustering algorithm is applied to the continuous spectral representation based on a vocabulary associated with the speech signal.

6. A method according to any preceding claim, wherein: Generating the plurality of changed content units includes one or more of: changing a content unit in the plurality of content units; adding a content unit to the plurality of content units; or deleting a content unit from the plurality of content units.

7. A method according to any preceding claim, wherein: The speech signal is associated with the speaker, wherein the method further comprises: Based on the speech signal, the speech characteristics of the speaker are generated.

8. A method according to any preceding claim, wherein: Generating the plurality of changed content units is based on a sequence-to-sequence model.

9. The method according to claim 8, wherein: The sequence-to-sequence model includes: an encoder shared among a plurality of source emotions including the source emotion; and a decoder shared among a plurality of target emotions including the target emotion.

10. The method according to claim 8, wherein: The sequence-to-sequence model comprises: a plurality of encoders respectively dedicated to a plurality of source emotions including the source emotions; and wherein the sequence-to-sequence model comprises: a plurality of decoders respectively dedicated to a plurality of target emotions including the target emotions.

11. The method according to claim 8, wherein: The sequence-to-sequence model comprises: an encoder shared among a plurality of source emotions including the source emotion; and wherein the sequence-to-sequence model comprises: a plurality of decoders respectively dedicated to a plurality of target emotions including the target emotion.

12. One or more computer-readable non-transitory storage media containing software operable when executed to perform a method according to any preceding claim.

13. A system comprising: one or more processors; and a non-volatile memory coupled to the one or more processors, the non-volatile memory comprising instructions executable by the one or more processors, the one or more processors being operable to perform a method according to any one of claims 1 to 11 when executing the instructions.

Images