Emotional voice conversion model training method and emotional voice conversion method

Abstract

The invention provides a training method of an emotional voice conversion model and an emotional voice conversion method, which are applied to the technical field of voice signal processing, and the training method comprises the following steps: obtaining an emotional voice training set which comprises a plurality of emotional voice samples; for each emotional voice sample, processing the emotional voice sample by using a content coding network to obtain a content coding feature, the content coding feature representing voice content corresponding to the emotional voice sample and key information related to the speaker; processing the emotional voice sample by using an emotional extraction network to obtain an emotional extraction feature, the emotional extraction feature representing an emotional corresponding to the emotional voice sample; processing the content coding features and the emotion extraction features by using a decoding vocoder to obtain predicted emotion voice information; and iteratively adjusting network parameters of the initial emotional voice conversion model according to the emotional voice samples, the content coding features and the emotional extraction features to obtain a trained target emotional voice conversion model.

Description

Technical Field

[0001] The present application relates to the field of speech signal processing technology, and more specifically, to a training method for an emotional speech conversion model, an emotional speech conversion method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As the most direct channel for communication between people, voice not only transmits knowledge but also conveys thoughts and emotions. Beyond basic information, voice also contains rich paralinguistic information, such as the speaker's identity, emotion, and tone. Currently, voice is widely used in human-computer interaction, becoming a new form of communication. Smart homes, voice assistants, and online courses all use voice commands to capture key information from speech, controlling devices to perform tasks or complete specific instructions. Voice can also be adjusted to conceal negative emotions in different social situations.

[0003] While robot-generated speech currently achieves a high level of clarity, it still lags behind human speech in terms of naturalness and emotional expression. This limitation makes it difficult for robots to express emotional feedback, thus hindering further improvements in user experience and satisfaction. This situation highlights a key challenge facing the field of human-computer interaction: how to ensure speech intelligibility while improving the naturalness and emotional expressiveness of robots' speech, thereby achieving interactions that more closely resemble human communication. Summary of the Invention

[0004] In view of this, the present application provides a training method for an emotional speech conversion model, an emotional speech conversion method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] One aspect of the present application provides a method for training an emotion-speech conversion model, comprising:

[0006] Acquire an emotional speech training set, wherein the emotional speech training set includes a plurality of emotional speech samples;

[0007] For each of the emotional speech samples, the emotional speech sample is processed using a content coding network to obtain a content coding feature, where the content coding feature represents key information related to the speech content and speaker corresponding to the emotional speech sample;

[0008] Processing the emotional speech sample using an emotion extraction network to obtain an emotion extraction feature, wherein the emotion extraction feature represents the emotion corresponding to the emotional speech sample;

[0009] Processing the content encoding features and the emotion extraction features using a decoding vocoder to obtain predicted emotional speech information, wherein the initial emotional speech conversion model includes the content encoding network, the emotion extraction network, and the decoding vocoder;

[0010] Based on the emotional speech samples, the content encoding features, and the emotion extraction features, the network parameters of the initial emotional speech conversion model are iteratively adjusted to obtain a trained target emotional speech conversion model.

[0011] Another aspect of the present application provides an emotional speech conversion method, comprising:

[0012] Obtaining voice information to be converted and emotional voice information;

[0013] The above-mentioned voice information to be converted and the above-mentioned emotional voice information are input into the target emotional voice conversion model, and the emotional voice conversion information is output, wherein the above-mentioned emotional voice conversion information represents the voice information formed by the emotion of the above-mentioned emotional voice information and the content of the above-mentioned voice information to be converted.

[0014] Another aspect of the present application provides a training device for an emotion-speech conversion model, comprising:

[0015] A first acquisition module is used to acquire an emotional speech training set, wherein the emotional speech training set includes a plurality of emotional speech samples;

[0016] A content coding module is configured to process each of the emotional speech samples using a content coding network to obtain content coding features, wherein the content coding features represent key information related to the speech content and speaker corresponding to the emotional speech sample;

[0017] An emotion extraction module is used to process the emotion speech sample using an emotion extraction network to obtain emotion extraction features, wherein the emotion extraction features represent the emotion corresponding to the emotion speech sample;

[0018] A decoding module, configured to process the content encoding features and the emotion extraction features using a decoding vocoder to obtain predicted emotional speech information, wherein the initial emotional speech conversion model includes the content encoding network, the emotion extraction network, and the decoding vocoder;

[0019] The adjustment module is used to iteratively adjust the network parameters of the initial emotional speech conversion model according to the emotional speech sample, the content encoding feature and the emotion extraction feature to obtain a trained target emotional speech conversion model.

[0020] Another aspect of the present application provides an emotional speech conversion device, comprising:

[0021] The second acquisition module is used to acquire the voice information to be converted and the emotional voice information;

[0022] The speech conversion module is used to input the above-mentioned speech information to be converted and the above-mentioned emotional speech information into the target emotional speech conversion model, and output emotional speech conversion information, wherein the above-mentioned emotional speech conversion information represents the speech information formed by the emotion of the above-mentioned emotional speech information and the content of the above-mentioned speech information to be converted.

[0023] Another aspect of the present application provides an electronic device, comprising:

[0024] one or more processors;

[0025] a memory for storing one or more programs,

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0027] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0028] Another aspect of the present application provides a computer program product, which includes computer-executable instructions. When the instructions are executed, the instructions are used to implement the method described above.

[0029] According to the embodiments of the present application, content coding features and emotion extraction features are extracted from emotional speech samples through a content coding network and an emotion extraction network respectively. Then, a decoding vocoder is used to process the content coding features and emotion extraction features to obtain predicted emotional speech information. Based on the emotional speech samples, content coding features, and emotion extraction features, the network parameters of the initial emotional speech conversion model are adjusted to obtain a target emotional speech conversion model. By adopting a content coding network with a variational encoder VAE structure to decouple language content from emotion, more accurate content coding features are obtained. Combined with the emotion extraction features extracted by the emotion extraction network, the fineness of the reconstruction of the predicted emotional speech information can be enhanced, thereby improving the quality of the predicted emotional speech information and the emotional expression ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0031] Figure 1 An exemplary system architecture to which an emotional speech conversion model training method or an emotional speech conversion method can be applied according to an embodiment of the present application is shown;

[0032] Figure 2 A flowchart of a method for training an emotional speech conversion model according to an embodiment of the present application is shown;

[0033] Figure 3 A schematic diagram of the model structure of the target emotion speech conversion model according to an embodiment of the present application is shown;

[0034] Figure 4 A flowchart of the emotional speech conversion method according to an embodiment of the present application is shown;

[0035] Figure 5 A block diagram of a training device for an emotional speech conversion model according to an embodiment of the present application is shown;

[0036] Figure 6 A block diagram of an emotional speech conversion device according to an embodiment of the present application is shown;

[0037] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0039] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0040] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0041] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0042] Emotional speech conversion technology aims to preserve the content and speaker identity while modulating emotion. This technology can transform a sentence that is supposed to express a neutral emotion into a different emotion, such as happiness, anger, or sadness, thereby enhancing the emotional expressiveness of the speech. For emotion-related research, such as emotional speech recognition and emotional speech conversion, extracting deep emotional features is particularly important. The quality of emotional feature extraction determines the effectiveness of the model.

[0043] For feature selection, most related technologies choose to use traditional acoustic features as input, such as fundamental frequency features F0, Mel-frequency cepstral coefficients MFCC, etc., and then further extract emotion-related embeddings through an encoder composed of a convolutional neural network CNN and a long short-term memory network LSTM. Although this extraction method can obtain emotional features and thus complete the emotion conversion task, the obtained features may be unstable, resulting in the retention of other information, such as the identity of the speaker, which will lead to incomplete model disentanglement and ultimately affect the degree of emotional expression of the converted speech.

[0044] In view of this, an embodiment of the present application provides a training method for an emotion speech conversion model and an emotion speech conversion method, the training method including obtaining an emotion speech training set, wherein the emotion speech training set includes multiple emotion speech samples; for each emotion speech sample, using a content coding network to process the emotion speech sample to obtain content coding features, the content coding features represent key information related to the speech content and speaker corresponding to the emotion speech sample; using an emotion extraction network to process the emotion speech sample to obtain emotion extraction features, wherein the emotion extraction features represent the emotion corresponding to the emotion speech sample; using a decoding vocoder to process the content coding features and the emotion extraction features to obtain predicted emotion speech information, the initial emotion speech conversion model including a content coding network, an emotion extraction network and a decoding vocoder; according to the emotion speech sample, the content coding features and the emotion extraction features, iteratively adjusting the network parameters of the initial emotion speech conversion model to obtain a trained target emotion speech conversion model.

[0045] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0046] Figure 1 FIG1 shows an exemplary system architecture 100 to which the training method of the emotional speech conversion model or the emotional speech conversion method according to an embodiment of the present application can be applied. It should be noted that, Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0047] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0048] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0049] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0050] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0051] It should be noted that the training method of the emotional speech conversion model or the emotional speech conversion method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the training device of the emotional speech conversion model or the emotional speech conversion device provided in the embodiment of the present application can generally be set in the server 105. The training method of the emotional speech conversion model or the emotional speech conversion method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device of the emotional speech conversion model or the emotional speech conversion device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the training method of the emotion speech conversion model or the emotion speech conversion method provided in the embodiment of the present application may also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be performed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the training device of the emotion speech conversion model or the emotion speech conversion device provided in the embodiment of the present application may also be provided in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may be provided in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0053] Figure 2 A flowchart of a method for training an emotional speech conversion model according to an embodiment of the present application is shown.

[0054] like Figure 2 As shown, the training method of the emotion speech conversion model includes operations S201 to S205.

[0055] In operation S201, an emotional speech training set is obtained, wherein the emotional speech training set includes a plurality of emotional speech samples;

[0056] In operation S202, for each emotional speech sample, the emotional speech sample is processed using a content coding network to obtain a content coding feature, where the content coding feature represents key information related to the speech content and speaker corresponding to the emotional speech sample;

[0057] In operation S203, the emotional speech sample is processed using an emotion extraction network to obtain an emotion extraction feature, wherein the emotion extraction feature represents the emotion corresponding to the emotional speech sample;

[0058] In operation S204, the content encoding features and the emotion extraction features are processed by a decoding vocoder to obtain predicted emotional speech information, wherein the initial emotional speech conversion model includes a content encoding network, an emotion extraction network, and a decoding vocoder;

[0059] In operation S205 , network parameters of the initial emotional speech conversion model are iteratively adjusted according to the emotional speech sample, the content encoding features, and the emotion extraction features to obtain a trained target emotional speech conversion model.

[0060] According to an embodiment of the present application, different emotional speech samples may be speech segments with different emotions, such as happiness, anger, rage, sadness, fear, etc.

[0061] According to an embodiment of the present application, for each emotional speech sample, the emotional speech sample is first input into a content coding network to extract the speech content and key information related to the speaker in the emotional speech sample. For example, if an emotional speech sample is user A's speech content "Hello, I am Zhang San", the content coding network can extract the speech content "Hello, I am Zhang San" and also extract information unrelated to the emotion of user A, such as timbre and speaking speed. The content coding network adopts a variational encoder (VAE) structure.

[0062] According to an embodiment of the present application, the emotional speech sample is input into an emotion extraction network for processing to extract the emotion contained in the emotional speech sample, that is, the emotion extraction feature. For example, the emotion extraction feature of user A can be identified when user A is in a happy mood (or other emotions) when saying "Hello, I am Zhang San".

[0063] It should be noted that the inputs of the content encoding network and the emotion extraction network can be the same emotional speech sample or different emotional speech samples.

[0064] According to an embodiment of the present application, the identified content coding features and emotion extraction features are input into a decoding vocoder for fusion to obtain predicted emotional speech information, which is the speech content read aloud with the emotion corresponding to the identified emotion extraction features.

[0065] According to the embodiments of the present application, content coding features and emotion extraction features are extracted from emotional speech samples through a content coding network and an emotion extraction network respectively. Then, a decoding vocoder is used to process the content coding features and emotion extraction features to obtain predicted emotional speech information. Based on the emotional speech samples, content coding features, and emotion extraction features, the network parameters of the initial emotional speech conversion model are adjusted to obtain a target emotional speech conversion model. By adopting a content coding network with a variational encoder VAE structure to decouple language content from emotion, more accurate content coding features are obtained. Combined with the emotion extraction features extracted by the emotion extraction network, the fineness of the reconstruction of the predicted emotional speech information can be enhanced, thereby improving the quality of the predicted emotional speech information and the emotional expression ability.

[0066] Figure 3 A schematic diagram of the model structure of the target emotion speech conversion model according to an embodiment of the present application is shown.

[0067] According to the embodiments of the present application, Figure 3 As shown, the emotional speech sample is processed by a content coding network to obtain content coding features, including: using an initial vocoder to perform feature extraction processing on the emotional speech sample to obtain fundamental frequency features and harmonic spectrum features; performing feature conversion processing on the harmonic spectrum features to obtain Mel-frequency cepstral coefficients; using an emotion-independent encoder to process the first input feature to obtain content coding features, wherein the first input feature includes fundamental frequency features and Mel-frequency cepstral coefficients.

[0068] According to an embodiment of the present application, the initial vocoder may be a vocoder based on a source-filter model, such as a WORLD vocoder, which describes the speech generation process as a process in which a series of pulse signals (or white noise) enter a linear time-invariant system.

[0069] According to an embodiment of the present application, for each emotional speech sample, an initial vocoder is used to extract the fundamental frequency feature F0, the harmonic spectrum feature SP, and the aperiodic signal AP from the emotional speech sample. Based on this, the extracted harmonic spectrum feature SP is further converted into Mel-frequency cepstral coefficients (MCEPs) through a specific conversion process.

[0070] According to an embodiment of the present application, the emotion-independent encoder adopts a variational autoencoder VAE structure, and the comprehensive feature vector obtained by merging the fundamental frequency feature F0 and the Mel-frequency cepstral coefficients MCEPs is used as input, and the content coding features related to the speech content and the speaker are extracted through the emotion-independent encoder.

[0071] According to the embodiments of the present application, Figure 3 As shown, the emotion-independent encoder includes L first convolutional normalization layers, which include the first convolutional layer and the first instance normalization layer, and L≥1.

[0072] According to an embodiment of the present application, the first input feature is processed using an emotion-independent encoder to obtain a content coding feature, including: for the i-th first convolutional normalization layer, the second input feature is convolved using the i-th first convolutional layer to obtain the i-th first convolutional feature, wherein, when i=1, the second input feature includes the first input feature, and when i≠1 and i is greater than or equal to 2, the second input feature includes the i-1th first normalized feature output by the i-1th first convolutional normalization layer; the i-th first convolutional feature is processed using the i-th first instance normalization layer to obtain the i-th first normalized feature, wherein, when i=L, the i-th first normalized feature represents the content coding feature.

[0073] According to an embodiment of the present application, for the first first convolution normalization layer, the first first convolution layer performs convolution processing on the fundamental frequency feature F0 and the Mel-cephalometric coefficients MCEPs to obtain the first first convolution feature, and the first first convolution feature is processed using the first first instance normalization layer to obtain the first first normalized feature.

[0074] According to an embodiment of the present application, for the i-th first convolution normalization layer from the 2nd to the L-1th, the i-th first convolution layer performs convolution processing on the i-1th first normalized feature to obtain the i-th first convolution feature, and the i-th first instance normalization layer is used to process the i-th first convolution feature to obtain the i-th first normalized feature.

[0075] According to an embodiment of the present application, for the Lth first convolution normalization layer, the Lth first convolution layer performs convolution processing on the L-1th first normalized feature to obtain the Lth first convolution feature, and the Lth first instance normalization layer is used to process the Lth first convolution feature to obtain the Lth first normalized feature. The Lth first normalized feature is also the content coding feature.

[0076] According to the embodiments of the present application, Figure 3 As shown, using the emotion extraction network to process the emotion speech sample to obtain the emotion extraction feature includes: using the speech emotion recognition base model to process the emotion speech sample to obtain the initial emotion feature; using the emotion extractor to process the initial emotion feature to obtain the emotion extraction feature.

[0077] According to an embodiment of the present application, the speech emotion recognition base model refers to the Emotion2vec model, which uses a self-supervised pre-training method and is trained on 262 hours of open-source emotion data. It uses an online distillation paradigm and contains two networks: a teacher network and a student network. These two networks have the same architecture, both including a feature extractor composed of a multi-layer convolutional neural network and a backbone network composed of a multi-layer Transformer. During the pre-training process, the model also combines sentence-level loss and frame-level loss to better capture emotional information.

[0078] According to an embodiment of the present application, the initial emotional features extracted by the speech emotion recognition base model are input into the emotion extractor. The emotion extractor can effectively extract the emotional features in the emotional speech sample by combining the convolutional layer, the gated recurrent network (GRU) and the feedforward neural network, and work together with the emotion-independent encoder to ensure that the emotional information is effectively separated from the non-emotional features such as the content of the speech and the identity of the speaker. Thus, the emotion extractor can output the emotion extraction features in the emotional speech sample.

[0079] According to the embodiments of the present application, Figure 3 As shown, the training method of the emotional speech conversion model also includes: using multiple linear layers to process the emotional extraction features to obtain emotional linear features; using the emotional classification network to perform emotional classification processing on the emotional linear features to obtain the emotional classification results of the emotional speech samples, wherein the initial emotional speech conversion model also includes the emotional classification network.

[0080] According to an embodiment of the present application, the emotion classification network can be any type of multi-classification network, such as a decision tree model, a support vector machine, etc. In the embodiment of the present application, the number of categories of the multi-classification network can be determined according to the number of emotion categories. For example, when emotions include happiness, sadness, neutrality, and anger, a four-classification network can be used. The use of an emotion classifier can ensure the consistency and accuracy of emotional expression.

[0081] According to an embodiment of the present application, linear transformation is performed using N layers of linear layers, and the emotion classification network maps the emotional linear features output by the linear layer to different emotion categories based on the Softmax activation function, thereby accurately predicting the emotion category of the input audio (i.e., emotional speech sample), i.e., the emotion classification result.

[0082] According to the embodiments of the present application, the use of the emotion classification network not only effectively improves the accuracy of emotion speech conversion, but also ensures that the predicted emotion speech information after conversion is more consistent with the expected emotion while ensuring the naturalness and intelligibility of the speech.

[0083] According to an embodiment of the present application, an emotion extractor is used to process the initial emotion features to obtain emotion extraction features, including: using the second convolutional layer to process the initial emotion features to obtain emotion convolution features; using a gated recurrent network to process the emotion convolution features to obtain emotion loop features; and using a feedforward neural network to process the emotion loop features to obtain emotion extraction features.

[0084] In a specific embodiment, the emotion extractor can use two second convolutional layers to reduce the dimensionality of the extracted 768-dimensional initial emotion features from 512 to 128 dimensions. This is then combined with a gated recurrent neural network (GRU) and a feedforward neural network to further obtain deep emotion features, namely, emotion extraction features. This structured emotion extractor not only retains the emotional information in emotional speech samples but also precisely controls the extraction and expression of emotional features, thereby preventing emotions from interfering with the performance of other speech features.

[0085] According to an embodiment of the present application, a decoding vocoder is used to process content coding features and emotion extraction features to obtain predicted emotional speech information, including: using a speech decoder to downsample the content coding features and emotion extraction features to obtain emotional speech decoding features; using a channel convolutional network to process the emotional speech decoding features to obtain channel convolution features, wherein the channel convolutional network includes m third convolutional layers, wherein the channel dimension of the last third convolutional layer is smaller than the channel dimension of the first m-1 third convolutional layers; using a speech vocoder to process the emotional speech decoding features and the channel convolution features to obtain predicted emotional speech information.

[0086] According to an embodiment of the present application, the channel convolutional network is a post-processing network for speech recognition, which may be a Post Net network.

[0087] According to an embodiment of the present application, a speech decoder is used to downsample and fuse content coding features and emotion extraction features, thereby obtaining emotional speech decoding features.

[0088] In one specific embodiment, the Post Net network consists of five third-convolutional layers with a kernel size of k = (5×1), whose input is the output of the speech decoder. The first four third-convolutional layers have a channel dimension of 512, and batch norm and Tanh activation are applied to these layers. The last third-convolutional layer has a channel dimension of 25 channels to compress the output to the same size as the input. Finally, a speech vocoder is used to concatenate the output of the speech decoder with the result of the Post Net network processing to form the final speech representation, namely the predicted emotional speech information. This concatenation mechanism ensures that the output predicted emotional speech information not only matches the target emotion but also further optimizes intelligibility and naturalness.

[0089] According to the embodiments of the present application, Figure 3 As shown, the speech decoder includes M second convolutional normalization layers, and the second convolutional normalization layers include a third convolutional layer and a second instance normalization layer, where M≥1.

[0090] According to an embodiment of the present application, a speech decoder is used to downsample the content coding features and the emotion extraction features to obtain the emotional speech decoding features, including: for the j-th second convolution normalization layer, the j-th third convolution layer is used to convolve the third input feature to obtain the j-th second convolution feature, wherein, when j=1, the third input feature includes the content coding feature and the emotion extraction feature, and when j≠1 and j is greater than or equal to 2, the input feature includes the j-1-th second normalized feature output by the j-1-th second convolution normalization layer; the j-th second convolution feature is processed by the j-th second instance normalization layer to obtain the j-th second normalized feature, wherein, when j=M, the j-th second convolution feature represents the emotional speech decoding feature.

[0091] According to an embodiment of the present application, the speech decoder effectively integrates the emotion representation (i.e., emotion extraction feature) with the speech content (i.e., content coding feature) and speaker information (such as the timbre in the content coding feature), and reconstructs multiple features through the second convolutional layer and the second instance normalization layer (IN), thereby generating a speech representation with the target emotion, i.e., the emotional speech decoding feature.

[0092] According to an embodiment of the present application, for the first second convolutional normalization layer, the first third convolutional layer performs convolution processing on the content encoding feature and the emotion extraction feature to obtain the first second convolutional feature, and the first second instance normalization layer processes the first second convolutional feature to obtain the first second normalized feature.

[0093] According to an embodiment of the present application, for any j-th second convolution normalization layer from the 2nd to the M-1th, the j-th third convolution layer performs convolution processing on the j-1th second normalized feature to obtain the j-th second convolution feature, and the j-th second instance normalization layer processes the j-th second convolution feature to obtain the j-th second normalized feature.

[0094] According to an embodiment of the present application, for the Mth second convolution normalization layer, the Mth third convolution layer performs convolution processing on the M-1th second normalized feature to obtain the Mth second convolution feature, and the Mth second instance normalization layer processes the Mth second convolution feature to obtain the Mth second normalized feature, which is the emotional speech decoding feature.

[0095] According to the embodiments of this application, the synergistic effect of the speech decoder and the channel convolutional network (i.e., the post-net network) significantly improves speech quality, enhances emotional expression, and effectively avoids speech distortion or unnaturalness that may occur during the emotion conversion process. The target emotion speech conversion model not only achieves high accuracy in emotional expression, but also maintains speech clarity and fluency.

[0096] According to an embodiment of the present application, the network parameters of the initial emotional speech conversion model are iteratively adjusted based on the emotional speech sample, the content encoding features, and the emotion extraction features to obtain a trained target emotional speech conversion model, including:

[0097] Calculate the target loss result based on the emotional speech sample, content encoding features and emotion extraction features;

[0098] The network parameters of the initial emotional speech conversion model are iteratively adjusted according to the target loss result to obtain the target emotional speech conversion model.

[0099] According to an embodiment of the present application, during the training process of the target emotional speech conversion model, iterative adjustment is performed using a self-supervised method. Specifically, the target loss result is calculated based on the emotional speech sample, content encoding features and emotion extraction features, and then parameter adjustment is performed based on the target loss result to obtain the target emotional speech conversion model.

[0100] According to an embodiment of the present application, a target loss result is calculated based on the emotional speech sample, content coding features and emotion extraction features, including: for each emotional speech sample, the feature mean and feature standard deviation are calculated based on multiple feature data in the content coding features; the divergence loss result is calculated based on the feature mean and feature standard deviation; the reconstruction loss result is calculated based on the emotional speech sample, the emotional speech decoding features output by the decoding vocoder, and the channel convolution features; the mutual information loss result is calculated based on the content coding features and the emotion extraction features; and the target loss result is generated based on at least one of the divergence loss result, the reconstruction loss result and the mutual information loss result.

[0101] When the initial emotional speech conversion model also includes an emotional classification network, the emotional consistency loss result is calculated based on the emotion type and emotional classification label corresponding to the predicted emotional speech information predicted by the emotional classification network, wherein the emotional speech training set also includes the emotional classification label corresponding to each emotional speech sample; the target loss result is generated based on the divergence loss result, the emotional consistency loss result, the reconstruction loss result and the mutual information loss result.

[0102] According to the embodiment of the present application, during the training process, the emotion-independent encoder uses KL divergence (Kullback-Leibler divergence) as a loss function to quantify the difference between the posterior distribution and the prior distribution, and guides the model learning by minimizing the loss. Therefore, for each emotional speech sample, the feature mean is calculated based on multiple feature data in the content encoding feature. and characteristic standard deviation , then according to the feature mean and characteristic standard deviation Calculate the divergence loss result , as shown in formula (1):

[0103] (1)

[0104] Where d represents the dimension of the content encoding feature z.

[0105] According to an embodiment of the present application, the channel convolutional network is used to further reconstruct fine details. For this part of the network, the corresponding reconstruction features As shown in formula (2):

[0106] (2)

[0107] Among them, x is the emotional speech decoding feature, is the channel convolution feature, It is the final reconstruction of MCEPs. Therefore, there is an additional reconstruction loss, which consists of two parts: one is the reconstruction loss of the original features and the decoder output features, and the other is the superposition of the original features, the decoder features and the Post net output. The reconstruction loss result is As shown in formula (3):

[0108] (3)

[0109] in, The embodiment of the present application improves the detail expression capability of the reconstructed predicted emotional speech information by combining the speech decoder and the post net network, while ensuring the effective preservation of the emotional information and speech content.

[0110] According to an embodiment of the present application, in order to more thoroughly separate emotional information from information unrelated to emotion, we use mutual information (MI) to reduce the correlation between content encoding features and emotion extraction features. The mutual information loss result is shown in formula (4):

[0111] (4)

[0112] Where c is the emotion extraction feature, z is the latent variable, K is the number of emotion categories, that is, the different emotion categories that the model needs to learn. T is the number of time steps or length of the sequence, which is usually related to the number of audio frames processed. and Represents the k-th category emotion code and the corresponding latent variable The value at time step t. represents the set of parameters in the model used to jointly model sentiment features and latent variables, 、 Refers to sentiment extraction features The variational approximation of the posterior distribution under a given latent variable z under sentiment category k or sentiment category l, where the latent variable is the content encoding feature.

[0113] According to an embodiment of the present application, the target loss result shown in formula (5) is constructed according to formulas (1) to (4):

[0114] (5)

[0115] in They are all weight coefficients, and their values ​​can be set according to actual needs.

[0116] According to the embodiment of the present application, when the initial emotional speech conversion model also includes an emotion classification network, a cross entropy loss function is designed as the emotion consistency loss to ensure that the emotion information is fully retained during the entire training process, while promoting model disentanglement and ensuring that the source of the emotion information comes from the emotion classification network. As shown in formula (6):

[0117] (6)

[0118] in, is the emotional classification label of the i-th emotional speech sample on the k-th emotional type, is the predicted probability of the emotion classification network that the i-th emotional speech sample belongs to the c-th emotion type, K is the total number of emotion types, and N is the number of emotional speech samples.

[0119] According to an embodiment of the present application, when the initial emotional speech conversion model also includes an emotion classification network, the target loss result is shown in formula (7):

[0120] (7)

[0121] in, Loss of sentiment consistency The weight coefficient of .

[0122] In a specific embodiment, each In the initial stage of the experiment, each The value range is set between 0.1 and 100, and the specific values ​​of multiple weight coefficients are finally determined, that is, , , , These values ​​not only ensure effective emotion separation but also preserve the audio quality of the converted speech.

[0123] In a specific embodiment, during the training process, the Adam optimizer can be used for parameter optimization, wherein the initial learning rate is set to 0.0005 and the momentum parameters are set to 0.9 and 0.999 to ensure the stability and convergence of the optimization process. The batch size used for model training is 128, and 128 frame segments are randomly selected from the emotional speech training set as a single emotional speech sample to enhance the model's generalization ability for diverse samples. The total number of training steps is set to 200,000 to ensure that the model can fully learn emotional features under large-scale training. The construction of the dataset strictly follows the principles of balance and diversity. For each speaker's data, 300 samples are selected from each emotional category to construct a training set, and 30 samples are retained as a test set to evaluate the model's generalization performance and emotional conversion effect.

[0124] Figure 4 A flowchart of an emotional speech conversion method according to an embodiment of the present application is shown.

[0125] like Figure 4 As shown, the emotional speech conversion method includes operations S401-S402.

[0126] In operation S401, voice information to be converted and emotional voice information are acquired.

[0127] In operation S402, the speech information to be converted and the emotional speech information are input into a target emotional speech conversion model, and the emotional speech conversion information is output, wherein the emotional speech conversion information represents speech information formed by the emotion of the emotional speech information and the content of the speech information to be converted.

[0128] According to an embodiment of the present application, the voice information to be converted and the emotional voice information may have different emotions. For example, the emotion of the voice information to be converted is classified as neutral, and the emotion of the emotional voice information is classified as happy.

[0129] According to an embodiment of the present application, the voice information to be converted and the emotional voice information are input into a trained target emotional voice conversion model to obtain emotional voice conversion information, wherein the emotional voice conversion information is the voice content in the voice information to be converted read aloud with emotion.

[0130] According to an embodiment of the present application, the target emotional speech conversion model uses a content coding network and an emotion extraction network to extract content coding features and emotion extraction features from the speech information to be converted and the emotional speech information, respectively. The decoding vocoder then processes the content coding features and emotion extraction features to obtain emotional speech conversion information. The target emotional speech conversion model of the present embodiment can enhance the sophistication of the reconstruction of emotional speech conversion information, improving the quality and emotional expressiveness of the emotional speech conversion information.

[0131] In a specific embodiment, the publicly available Emotional Speech Dataset (ESD) was used as the emotional speech training set. This dataset contains speech samples from four speakers, two male and two female. To ensure diversity and representativeness, this example selected the four emotion categories included in the dataset: anger, happiness, neutral, and sadness. Each speaker recorded 300 speech samples for each emotion, for a total of 4,800 speech samples, which are the emotional speech samples.

[0132] First, the WORLD vocoder (i.e., the initial vocoder) extracts basic acoustic features such as the fundamental frequency feature F0, the harmonic spectrum feature SP (spectral envelope), and the non-periodic signal AP (pitch envelope) from each speech. In addition, the Emotion2vec model is used to perform sentiment analysis on each sentence and extract the 768-dimensional initial sentiment features of each speech. These sentiment features play a vital role in subsequent processing. In terms of feature processing, this embodiment converts the extracted harmonic spectrum feature SP into Mel-frequency cepstral coefficients MCEPs for further analysis. Next, in order to enhance the expressive power of the model, MCEPs are concatenated with the F0 feature to obtain a comprehensive feature vector containing more speech information. By calculating the feature mean of these comprehensive features and characteristic standard deviation , each feature is normalized to ensure the uniformity of the feature’s numerical range, thereby improving the stability and effect of subsequent processing and model training.

[0133] The F0 and MCEP values ​​are used as input to the emotion-independent encoder, and the initial emotion features are used as input to the emotion extractor. The emotion-independent encoder consists of six first convolutional normalization layers. The activation of the emotion-independent encoder is used to guide the removal of emotional information. The emotion extractor, consisting of two second convolutional layers, a GRU, and a feedforward neural network, aims to extract key emotion-extracting features. These extracted emotion-extracting features are then used in a sentiment classification network to predict the emotion category. This is then matched to the actual emotion category of each sentence, allowing for better control of emotional expression throughout the training process.

[0134] The content encoding features and the emotion extraction features are combined through a decoder. The speech decoder mainly consists of M second convolutional normalization layers to better reconstruct the speech signal. The reconstructed emotional speech decoding features are fed into a channel convolutional network and a speech vocoder to further synthesize and predict emotional speech information.

[0135] The emotional speech conversion information obtained through conversion using the target emotional speech conversion model and related technologies of this embodiment was evaluated both subjectively and objectively. In the subjective evaluation phase, 10 participants (5 men and 5 women) were invited to rate the emotional speech conversion information obtained using different methods. Participants used a 5-point mean opinion score (MOS) and the XAB test to assess speech quality, emotional similarity, and intensity of emotional expression. The test set used for evaluation contained 360 utterances from four speakers (2 men and 2 women) in three different emotional states, with 120 utterances per state. For objective evaluation, this embodiment used Mel-Cepstral Distortion (MCD) and F0 Root Mean Square Error (F0-RMSE) as evaluation metrics, using the same test set to ensure consistency. These two metrics are widely used in the field of speech conversion, particularly in emotional speech conversion. By measuring the quality of the reconstruction of the speech spectrum and fundamental frequency, they can assess the similarity between the converted speech and the target speech.

[0136] Specifically, MCD primarily reflects the degree of distortion in the speech signal's spectral characteristics, while F0-RMSE quantifies the difference between the converted fundamental frequency curve and the target fundamental frequency curve. Lower values ​​for these two metrics indicate that the converted speech is closer to the target speech in terms of spectral characteristics and fundamental frequency contour, thus indicating better emotional conversion. Table 1 shows that the speech output by the target emotional speech conversion model of this embodiment exhibits good emotional conversion performance across these two metrics.

[0137] Table 1 MCD[dB] and F0-RMSE[hZ] results

[0138] Among them, N2A, N2H, and N2S are different validation sets, which are used to verify the superiority of the method of this embodiment compared with related technologies such as OneShot EVC, VAW-GAN, and ZEST models.

[0139] To assess speech quality, emotional similarity, and emotional expression, this example designed two subjective evaluation experiments. Speech quality was evaluated using the Mean Opinion Score (MOS). After listening to the converted speech, participants rated the speech quality on a scale of 1 to 5 (1: [Poor], 2: [Poor], 3: [Fair], 4: [Good], 5: [Excellent]). The MOS score was calculated by averaging all listeners' ratings. The results in Table 2 show that the target emotional speech conversion model proposed in this example significantly outperforms the baseline model and other related technologies in speech quality, achieving a higher MOS score.

[0140] Table 2 Naturalness and intelligibility MOS scores

[0141] Among them, OneShot EVC, VAW-GAN, ZEST, and Reference are all commonly used speech conversion models in related technologies.

[0142] Figure 5 A block diagram of a training device for an emotional speech conversion model according to an embodiment of the present application is shown.

[0143] like Figure 5 As shown, the training device 500 for the emotion speech conversion model includes a first acquisition module 510, a content encoding module 520, an emotion extraction module 530, a decoding module 540, and an adjustment module 550.

[0144] The first acquisition module 510 is used to acquire an emotional speech training set, wherein the emotional speech training set includes a plurality of emotional speech samples.

[0145] The content coding module 520 is used to process each emotional speech sample using a content coding network to obtain content coding features. The content coding features represent key information related to the speech content and speaker corresponding to the emotional speech sample.

[0146] The emotion extraction module 530 is used to process the emotional speech sample using the emotion extraction network to obtain emotion extraction features, wherein the emotion extraction features represent the emotion corresponding to the emotional speech sample.

[0147] The decoding module 540 is used to process the content coding features and the emotion extraction features using a decoding vocoder to obtain predicted emotional speech information. The initial emotional speech conversion model includes a content coding network, an emotion extraction network and a decoding vocoder.

[0148] The adjustment module 550 is used to iteratively adjust the network parameters of the initial emotional speech conversion model according to the emotional speech samples, content encoding features and emotion extraction features to obtain a trained target emotional speech conversion model.

[0149] According to the embodiments of the present application, content coding features and emotion extraction features are extracted from emotional speech samples through a content coding network and an emotion extraction network respectively. Then, a decoding vocoder is used to process the content coding features and emotion extraction features to obtain predicted emotional speech information. Based on the emotional speech samples, content coding features, and emotion extraction features, the network parameters of the initial emotional speech conversion model are adjusted to obtain a target emotional speech conversion model. By adopting a content coding network with a variational encoder VAE structure to decouple language content from emotion, more accurate content coding features are obtained. Combined with the emotion extraction features extracted by the emotion extraction network, the fineness of the reconstruction of the predicted emotional speech information can be enhanced, thereby improving the quality of the predicted emotional speech information and the emotional expression ability.

[0150] According to an embodiment of the present application, the content encoding module 520 includes a feature extraction unit, a feature conversion unit, and a first obtaining unit.

[0151] The feature extraction unit is used to perform feature extraction processing on the emotional speech sample using the initial vocoder to obtain fundamental frequency features and harmonic spectrum features.

[0152] The feature conversion unit is used to perform feature conversion processing on the harmonic spectrum features to obtain Mel-frequency cepstral coefficients.

[0153] The first obtaining unit is used to process the first input feature using the emotion-independent encoder to obtain a content coding feature, wherein the first input feature includes a fundamental frequency feature and a Mel-frequency cepstral coefficient.

[0154] According to an embodiment of the present application, the emotion-independent encoder includes L first convolutional normalization layers, the first convolutional normalization layers include a first convolutional layer and a first instance normalization layer, and L≥1.

[0155] According to an embodiment of the present application, the first obtaining unit includes a first obtaining sub-unit and a second obtaining sub-unit.

[0156] The first obtaining subunit is used to perform convolution processing on the second input feature using the i-th first convolutional normalization layer for the i-th first convolutional normalization layer to obtain the i-th first convolutional feature, wherein when i=1, the second input feature includes the first input feature, and when i≠1 and i is greater than or equal to 2, the second input feature includes the i-1th first normalized feature output by the i-1th first convolutional normalization layer.

[0157] The second obtaining subunit is used to process the i-th first convolution feature using the i-th first instance normalization layer to obtain the i-th first normalized feature, wherein when i=L, the i-th first normalized feature represents the content encoding feature.

[0158] According to an embodiment of the present application, the emotion extraction module 530 includes a first processing unit and a second processing unit.

[0159] The first processing unit is used to process the emotional speech sample using the speech emotion recognition base model to obtain initial emotional features.

[0160] The second processing unit is used to process the initial emotion features using the emotion extractor to obtain emotion extraction features.

[0161] According to an embodiment of the present application, the training device 500 further includes a linear module and a classification module.

[0162] The linear module is used to process the sentiment extraction features using multiple linear layers to obtain sentiment linear features.

[0163] The classification module is used to perform emotion classification processing on the emotion linear features using the emotion classification network to obtain the emotion classification results of the emotion speech samples, wherein the initial emotion speech conversion model also includes the emotion classification network.

[0164] According to an embodiment of the present application, the second processing unit includes a first convolution subunit, a first processing subunit, and a second processing subunit.

[0165] The first convolution subunit is used to obtain the emotional convolution feature by using the second convolution layer to perform the initial emotional feature.

[0166] The first processing subunit is used to process the sentiment convolution feature using a gated recurrent network to obtain the sentiment recurrent feature.

[0167] The second processing subunit is used to process the emotion loop feature using a feedforward neural network to obtain emotion extraction features.

[0168] According to an embodiment of the present application, the decoding module 540 includes a sampling unit, a convolution unit, and a second obtaining unit.

[0169] The sampling unit is used to downsample the content coding features and the emotion extraction features using a speech decoder to obtain the emotion speech decoding features.

[0170] The convolution unit is used to process the emotional speech decoding features using a channel convolution network to obtain channel convolution features, wherein the channel convolution network includes m third convolution layers, wherein the channel dimension of the last third convolution layer is smaller than the channel dimension of the first m-1 third convolution layers.

[0171] The second obtaining unit is used to process the emotional speech decoding features and channel convolution features using a speech vocoder to obtain predicted emotional speech information.

[0172] According to an embodiment of the present application, the speech decoder includes M second convolutional normalization layers, the second convolutional normalization layers include a third convolutional layer and a second instance normalization layer, and M≥1.

[0173] According to an embodiment of the present application, the sampling unit includes a second convolution subunit and a normalization subunit.

[0174] The second convolution subunit is used to perform convolution processing on the third input feature using the j-th second convolutional normalization layer for the j-th second convolutional normalization layer to obtain the j-th second convolutional feature, wherein when j=1, the third input feature includes content encoding features and sentiment extraction features, and when j≠1 and j is greater than or equal to 2, the input feature includes the j-1-th second normalized feature output by the j-1-th second convolutional normalization layer.

[0175] The normalization subunit is used to process the jth second convolution feature using the jth second instance normalization layer to obtain the jth second normalized feature, wherein when j=M, the jth second convolution feature represents the emotional speech decoding feature.

[0176] According to an embodiment of the present application, the adjustment module 550 includes a calculation unit and an adjustment unit.

[0177] The calculation unit is used to calculate the target loss result according to the emotional speech sample, content encoding features and emotion extraction features.

[0178] The adjustment unit is used to iteratively adjust the network parameters of the initial emotional speech conversion model according to the target loss result to obtain the target emotional speech conversion model.

[0179] According to an embodiment of the present application, the computing unit includes a first computing subunit, a second computing subunit, a third computing subunit, a fourth computing subunit, a first generating subunit, a fifth computing subunit, and a second generating subunit.

[0180] The first calculation subunit is used to calculate, for each emotional speech sample, a feature mean and a feature standard deviation according to multiple feature data in the content coding feature.

[0181] The second calculation subunit is used to calculate the divergence loss result according to the feature mean and the feature standard deviation.

[0182] The third calculation subunit is used to calculate the reconstruction loss result according to the emotional speech sample, the emotional speech decoding features output by the decoding vocoder, and the channel convolution features.

[0183] The fourth calculation subunit is used to calculate the mutual information loss result based on the content encoding features and the sentiment extraction features.

[0184] The first generating subunit is used to generate a target loss result according to at least one of the divergence loss result, the reconstruction loss result and the mutual information loss result.

[0185] The fifth computing subunit is used to calculate the emotional consistency loss result based on the emotion type and emotion classification label corresponding to the predicted emotional speech information predicted by the emotion classification network when the initial emotional speech conversion model also includes an emotion classification network, wherein the emotional speech training set also includes an emotion classification label corresponding to each emotional speech sample.

[0186] The second generation subunit is used to generate a target loss result based on the divergence loss result, the sentiment consistency loss result, the reconstruction loss result and the mutual information loss result.

[0187] Figure 6 A block diagram of an emotional speech conversion device according to an embodiment of the present application is shown.

[0188] like Figure 6 As shown, the emotional speech conversion device 600 includes a second acquisition module 610 and a speech conversion module 620 .

[0189] The second acquisition module 610 is used to acquire the voice information to be converted and the emotional voice information.

[0190] The speech conversion module 620 is used to input the speech information to be converted and the emotional speech information into the target emotional speech conversion model, and output the emotional speech conversion information, wherein the emotional speech conversion information represents the speech information formed by the emotion of the emotional speech information and the content of the speech information to be converted.

[0191] According to an embodiment of the present application, the target emotional speech conversion model uses a content coding network and an emotion extraction network to extract content coding features and emotion extraction features from the speech information to be converted and the emotional speech information, respectively. The decoding vocoder then processes the content coding features and emotion extraction features to obtain emotional speech conversion information. The target emotional speech conversion model of the present embodiment can enhance the sophistication of the reconstruction of emotional speech conversion information, improving the quality and emotional expressiveness of the emotional speech conversion information.

[0192] According to the embodiments of the present application, any number of modules, submodules, units, and subunits, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0193] It should be noted that the training device and the emotional speech conversion device of the emotional speech conversion model in the embodiments of the present application correspond to the training method and the emotional speech conversion method of the emotional speech conversion model in the embodiments of the present application. The description of the training device and the emotional speech conversion device of the emotional speech conversion model specifically refers to the training method and the emotional speech conversion method of the emotional speech conversion model, and will not be repeated here.

[0194] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0195] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0196] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0197] According to an embodiment of the present application, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0198] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0199] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0200] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0201] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present application are performed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0202] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0203] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0204] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The present application does not depart from the scope of the present application, and those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A training method for an emotional speech conversion model, characterized in that: include: Acquire an emotional speech training set, wherein the emotional speech training set includes a plurality of emotional speech samples; For each of the emotional speech samples, processing the emotional speech sample using a content coding network to obtain a content coding feature, wherein the content coding feature represents key information related to the speech content and speaker corresponding to the emotional speech sample; Processing the emotional speech sample using an emotion extraction network to obtain an emotion extraction feature, wherein the emotion extraction feature represents an emotion corresponding to the emotional speech sample; Processing the content encoding features and the emotion extraction features using a decoding vocoder to obtain predicted emotional speech information, wherein the initial emotional speech conversion model includes the content encoding network, the emotion extraction network and the decoding vocoder; The network parameters of the initial emotional speech conversion model are iteratively adjusted according to the emotional speech sample, the content encoding feature and the emotion extraction feature to obtain a trained target emotional speech conversion model.

2. The method according to claim 1, characterized in that The emotional speech sample is processed using a content coding network to obtain content coding features, including: Performing feature extraction processing on the emotional speech sample using an initial vocoder to obtain fundamental frequency features and harmonic spectrum features; Performing feature conversion processing on the harmonic spectrum features to obtain Mel-frequency cepstral coefficients; The first input feature is processed using an emotion-independent encoder to obtain the content coding feature, wherein the first input feature includes the fundamental frequency feature and the Mel-frequency cepstral coefficient.

3. The method according to claim 2, characterized in that The emotion-independent encoder includes L first convolutional normalization layers, where the first convolutional normalization layers include a first convolutional layer and a first instance normalization layer, and L≥1; The step of processing the first input feature using the emotion-independent encoder to obtain the content encoding feature includes: For the i-th first convolutional normalization layer, convolve the second input feature using the i-th first convolutional layer to obtain the i-th first convolutional feature, where when i=1, the second input feature includes the first input feature; when i≠1 and i is greater than or equal to 2, the second input feature includes the i-1th first normalized feature output by the i-1th first convolutional normalization layer; The i-th first convolutional feature is processed using the i-th first instance normalization layer to obtain the i-th first normalized feature, wherein when i=L, the i-th first normalized feature represents the content encoding feature.

4. The method according to claim 1, wherein The emotional speech sample is processed using an emotion extraction network to obtain emotion extraction features, including: Processing the emotional speech sample using a speech emotion recognition base model to obtain initial emotional features; Processing the initial emotion feature using an emotion extractor to obtain the emotion extraction feature; Wherein, the training method further includes: Processing the emotion extraction features using multiple linear layers to obtain emotion linear features; The emotion linear feature is subjected to emotion classification processing by using an emotion classification network to obtain an emotion classification result of the emotion speech sample, wherein the initial emotion speech conversion model also includes the emotion classification network.

5. The method according to claim 4, characterized in that Processing the initial emotion feature using an emotion extractor to obtain the emotion extraction feature includes: Using the second convolutional layer to process the initial emotional features, a sentiment convolution feature is obtained; Processing the sentiment convolutional features using a gated recurrent network to obtain sentiment recurrent features; The emotion cycle feature is processed using a feedforward neural network to obtain the emotion extraction feature.

6. The method according to claim 1, characterized in that Processing the content encoding features and the emotion extraction features using a decoding vocoder to obtain predicted emotional speech information includes: Downsampling the content encoding feature and the emotion extraction feature using a speech decoder to obtain an emotion speech decoding feature; Processing the emotional speech decoding features using a channel convolutional network to obtain channel convolutional features, wherein the channel convolutional network includes m third convolutional layers, wherein the channel dimension of the last third convolutional layer is smaller than the channel dimension of the first m-1 third convolutional layers; The emotional speech decoding feature and the channel convolution feature are processed by a speech vocoder to obtain the predicted emotional speech information.

7. The method according to claim 6, characterized in that The speech decoder includes M second convolutional normalization layers, each of which includes a third convolutional layer and a second instance normalization layer, where M≥1; The method of downsampling the content coding feature and the emotion extraction feature using a speech decoder to obtain an emotion speech decoding feature includes: For the j-th second convolutional normalization layer, convolution processing is performed on the third input feature using the j-th third convolutional layer to obtain the j-th second convolutional feature, wherein when j=1, the third input feature includes the content encoding feature and the emotion extraction feature, and when j≠1 and j is greater than or equal to 2, the input feature includes the j-1-th second normalized feature output by the j-1-th second convolutional normalization layer; The jth second convolution feature is processed using the jth second instance normalization layer to obtain the jth second normalized feature, wherein when j=M, the jth second convolution feature represents the emotional speech decoding feature.

8. The method according to claim 1, characterized in that Iteratively adjusting network parameters of the initial emotional speech conversion model according to the emotional speech sample, the content encoding feature, and the emotion extraction feature to obtain a trained target emotional speech conversion model, including: Calculating a target loss result according to the emotional speech sample, the content encoding feature, and the emotion extraction feature; The network parameters of the initial emotional speech conversion model are iteratively adjusted according to the target loss result to obtain the target emotional speech conversion model.

9. The method according to claim 8, characterized in that Calculating a target loss result according to the emotional speech sample, the content encoding feature, and the emotional extraction feature, including: For each of the emotional speech samples, calculating a feature mean and a feature standard deviation based on multiple feature data in the content coding feature; Calculating a divergence loss result according to the feature mean and the feature standard deviation; Calculating a reconstruction loss result based on the emotional speech sample, the emotional speech decoding features output by the decoding vocoder, and the channel convolution features; Calculating a mutual information loss result based on the content encoding feature and the emotion extraction feature; generating the target loss result according to at least one of the divergence loss result, the reconstruction loss result, and the mutual information loss result; In a case where the initial emotional speech conversion model further includes an emotional classification network, calculating an emotional consistency loss result according to the emotional type and emotional classification label corresponding to the predicted emotional speech information predicted by the emotional classification network, wherein the emotional speech training set further includes an emotional classification label corresponding to each of the emotional speech samples; The target loss result is generated according to the divergence loss result, the sentiment consistency loss result, the reconstruction loss result and the mutual information loss result.

10. A method for emotional speech conversion, characterized in that: include: Obtaining voice information to be converted and emotional voice information; Inputting the speech information to be converted and the emotional speech information into a target emotional speech conversion model, and outputting emotional speech conversion information, wherein the emotional speech conversion information represents speech information formed by the emotion of the emotional speech information and the content of the speech information to be converted; Wherein, the target emotional speech conversion model is trained by the method according to any one of claims 1 to 9.

Images