Emotional voice conversion method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of voice processing, and discloses an emotional voice conversion method and device, computer equipment and a storage medium, which are specifically suitable for voice conversion in the financial field and the medical pension field, and the method comprises the following steps: obtaining original audio and emotional control information; extracting a speaker feature vector corresponding to the original audio by using a preset identity holder; inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information; encoding plain text content in the original audio by using a preset content encoder to obtain a content encoding vector; and generating a converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content coding vector. The emotion control effect of the converted audio can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to an emotional speech conversion method, device, computer equipment and storage medium, which can be specifically applicable to speech conversion in the financial field and the medical and elderly care field. Background Art

[0002] Current emotional speech conversion systems are mainly based on autoencoders and sequence-to-sequence (seq2seq) frameworks. Although significant progress has been made in speech quality, the converted audio lacks emotional diversity and is difficult to meet the needs of realistic speech synthesis. To address this problem, some studies have attempted to incorporate an emotion intensity control module into the emotional speech conversion framework.

[0003] At present, when performing speech conversion, emotions are usually learned from reference audio, and then the learned emotions are added to the original audio, or emotional features are directly generated based on emotional tags, and the emotional features are superimposed on the original audio to achieve the purpose of speech emotion control. However, the original audio already carries emotions, and the above-mentioned method of the prior art is equivalent to directly superimposing or covering emotions on the basis of the emotions of the original audio, which does not consider the impact of the emotional fluctuations of the original audio itself, so it is difficult to guarantee the emotional control effect of the converted audio. Summary of the invention

[0004] The present invention provides an emotional speech conversion method, device, computer equipment and storage medium, which are mainly capable of ensuring the emotional control effect of the converted audio.

[0005] In a first aspect, a method for emotional speech conversion is provided, comprising:

[0006] Get raw audio and emotion control information;

[0007] Extracting a speaker feature vector corresponding to the original audio using a preset identity retainer;

[0008] Inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information;

[0009] Encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector;

[0010] Generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector.

[0011] In a second aspect, an emotional speech conversion device is provided, comprising:

[0012] An acquisition unit, used for acquiring original audio and emotion control information;

[0013] An extraction unit, used for extracting a speaker feature vector corresponding to the original audio using a preset identity retainer;

[0014] A mapping unit, used for inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information;

[0015] An encoding unit, used for encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector;

[0016] A generating unit is used to generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content coding vector.

[0017] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotional speech conversion method when executing the computer program.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned emotional speech conversion method are implemented.

[0019] The present application provides an emotional speech conversion method, device, computer equipment and storage medium. Compared with the prior art, the present application uses a preset identity holder to extract the speaker feature vector corresponding to the original audio, and inputs the emotion control information into the preset emotion mapper for emotion mapping to achieve emotion control in the speech conversion process. At the same time, the plain text content in the original audio is encoded using a preset content encoder to obtain a content encoding vector, and finally, according to the speaker feature vector, the emotion vector and the content encoding vector, the converted audio corresponding to the original audio is generated. Since the present application can use the preset content encoder to strip the content from the original audio when performing emotion control, and only encode the plain text content, the original emotion information can be removed from the original audio, and then combined with the emotion control information and the speaker characteristics to generate the converted audio, thereby ensuring that the converted audio is not affected by the emotional fluctuations of the original audio itself, and thus the emotion control effect can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of an application environment provided by an embodiment of the present application is shown.

[0022] Figure 2 A schematic diagram of a flow chart of an emotional speech conversion method provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of a process for generating converted audio provided by an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the overall architecture of voice conversion provided by an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram of the model training process provided in an embodiment of the present application is shown;

[0026] Figure 6 A schematic diagram of the overall architecture of the model training provided in the embodiment of the present application is shown;

[0027] Figure 7 A schematic diagram of the structure of an emotional speech conversion device provided in an embodiment of the present application is shown;

[0028] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The emotional speech conversion method provided by the embodiment of the present invention can be applied in Figure 1In the application environment, the client communicates with the server through the network. The server can obtain the original audio and emotion control information, and then use the preset identity holder to extract the speaker feature vector corresponding to the original audio, and input the emotion control information into the preset emotion mapper for emotion mapping to obtain the emotion vector corresponding to the emotion control information. At the same time, the plain text content in the original audio is encoded by the preset content encoder to obtain the content encoding vector, and finally the converted audio corresponding to the original audio is generated according to the speaker feature vector, the emotion vector and the content encoding vector. Since the embodiment of the present invention can use the preset content encoder to strip the content from the original audio when performing emotion control, and only encode the plain text content, the original emotion information can be removed from the original audio, and then combined with the emotion control information and the speaker characteristics to generate the converted audio, thereby ensuring that the converted audio is not affected by the emotion fluctuations of the original audio itself, and then the emotion control effect can be ensured. The above-mentioned speech conversion process of the embodiment of the present invention can be applicable to the financial field and the medical and elderly care field. The client of the embodiment of the present invention can be, but not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end of the embodiment of the present invention can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] See also Figure 2 As shown, Figure 2 A flow chart of the emotional speech conversion method provided by an embodiment of the present invention includes the following steps:

[0032] S10, obtaining original audio and emotion control information.

[0033] Among them, the emotion control information includes target emotion intensity and target emotion label. The target emotion label is the emotion type to be controlled, such as happiness, sadness, sadness and anger, etc. The target emotion intensity is the emotion intensity to be controlled, such as general happiness, moderate happiness and special happiness, etc.

[0034] For the embodiment of the present invention, the user can configure the relevant target emotion intensity and target emotion label, i.e., emotion control information, according to the actual emotion control needs. For example, the user hopes that the converted audio has a particularly happy emotion, so the target emotion label can be configured as happy and the target emotion intensity as particularly happy. At the same time, the original audio to be converted also needs to be obtained.

[0035] The embodiment of the present invention can control not only the emotion type of the converted speech but also the emotion intensity of the converted speech through the emotion control information.

[0036] S20. Using a preset identity retainer, extract a speaker feature vector corresponding to the original audio.

[0037] In the embodiment of the present invention, the original audio may be input into a preset identity retainer to extract a speaker feature vector, and the feature vector may be used to characterize who said the original audio, ie, the speaker feature.

[0038] S30: Input the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information.

[0039] Among them, the preset emotion mapper can map emotion intensity and emotion label into emotion vector, and can also map emotion vector into emotion intensity and emotion label, that is, through the preset emotion mapper, emotion intensity and emotion label can be mapped to emotion vector.

[0040] For an embodiment of the present invention, after obtaining emotion control information, that is, obtaining the target emotion intensity and the target emotion label, the target emotion intensity and the target emotion label are input into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information. The emotion vector can specifically be VAD, which is a two-dimensional representation of an emotion.

[0041] S40: Encode the plain text content in the original audio using a preset content encoder to obtain a content encoding vector.

[0042] According to the embodiment of the present invention, the phoneme sequence corresponding to the original audio is input into a preset content encoder, and the plain text content in the original audio is encoded to obtain a content encoding vector.

[0043] Since the embodiment of the present invention can utilize a preset content encoder to strip content from the original audio and only encode the plain text content, the original emotional information can be removed from the original audio, thereby avoiding direct emotional superposition on the basis of the original emotional information during emotional control, thereby ensuring the emotional control effect of the converted audio.

[0044] S50. Generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content coding vector.

[0045] The converted audio is audio that introduces emotion control.

[0046] In the embodiment of the present invention, after extracting the speaker feature vector, the emotion vector and the content coding vector, the converted audio corresponding to the original audio is generated based on the speaker feature vector, the emotion vector and the content coding vector. Figure 3 As shown, including:

[0047] S51, inputting the emotion vector and the content coding vector into a preset duration predictor for duration prediction and content expansion, to obtain a content expansion sequence carrying corresponding emotions.

[0048] In the embodiment of the present invention, since there is a certain time difference between the phonemes input into the preset content encoder and the audio, it is necessary to predict the duration of the content encoding vector, and at the same time, in order to perform emotion control, it is necessary to introduce the emotion vector. Specifically, the emotion vector and the content encoding vector are input into the preset duration predictor together to predict the duration corresponding to each data point of the content encoding vector, so as to perform content expansion based on the duration, and finally obtain a content expansion sequence carrying the corresponding emotion.

[0049] In some embodiments, in order to ensure the accuracy of the content extension sequence carrying the corresponding emotion, the embodiment of the present invention also introduces a preset emotion renderer, which first performs emotion rendering on the emotion vector and the content coding vector, and then performs duration prediction, such as Figure 4 As shown. For this process, the method includes: inputting the emotion vector and the content coding vector into a preset emotion renderer for emotion rendering to obtain a content coding vector carrying the corresponding emotion; inputting the content coding vector carrying the corresponding emotion and the content coding vector into a preset duration predictor for duration prediction and content expansion to obtain a content expansion sequence carrying the corresponding emotion.

[0050] The embodiment of the present invention utilizes the residual concept, that is, first using a preset emotion renderer to obtain a content coding vector carrying the corresponding emotion, and then inputting the content coding vector carrying the corresponding emotion and the content coding vector into a preset duration predictor for duration prediction and content expansion, thereby avoiding information loss and ensuring content authenticity.

[0051] S52: Generate converted audio corresponding to the original audio according to the content extension sequence carrying the corresponding emotion and the speaker feature vector.

[0052] For the embodiment of the present invention, in order to generate converted audio corresponding to the original audio based on the content extension sequence carrying the corresponding emotion and the speaker feature vector, step S52 specifically includes: inputting the content extension sequence carrying the corresponding emotion into the preset flow model for vector conversion to obtain the target feature vector; generating latent variables according to the speaker feature vector and the target feature vector; inputting the latent variables into the preset decoder for decoding, and outputting the converted audio. Among them, the preset flow model is a Flow model, which can be used for forward conversion and reverse conversion between vectors and content.

[0053] Furthermore, the embodiment of the present invention also provides a training method for a preset identity retainer, a preset emotion mapper, a preset duration predictor, and a preset content encoder, such as Figure 5 As shown, including:

[0054] S60: Obtain sample original audio and an emotion label corresponding to the sample original audio.

[0055] Among them, the emotion label is the emotion type to be controlled, such as happiness, sadness, sadness and anger.

[0056] For the embodiment of the present invention, in the model training stage, the sample original audio, the speed sequence corresponding to the sample original audio and the emotion label are used as input data.

[0057] S70, construct an initial identity retainer, an initial emotion mapper, an initial emotion evaluator, an initial content encoder, an initial duration predictor, an initial flow model, an initial decoder, an initial reconstruction classifier and an initial emotion classifier.

[0058] For the embodiment of the present invention, the parameters in the identity retainer, emotion mapper, emotion evaluator, content encoder, duration predictor, flow model, decoder, reconstruction classifier and emotion classifier are initialized to construct the initial identity retainer, initial emotion mapper, initial emotion evaluator, initial content encoder, initial duration predictor, initial flow model, initial decoder, initial reconstruction classifier and initial emotion classifier.

[0059] S80. Based on the original audio and the emotion labels corresponding to the sample original audio, the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier are iteratively trained.

[0060] For the embodiment of the present invention, when performing model training, step S80 specifically includes: using the initial identity retainer to extract the speaker sample feature vector corresponding to the original audio; inputting the sample original audio and the emotion label into the initial emotion evaluator for emotion evaluation to obtain a sample emotion vector; determining sample latent variables based on the speaker sample feature vector and the sample emotion vector; inputting the sample latent variables into the initial decoder for decoding to obtain converted sample audio; inputting the converted sample audio into the initial reconstruction classifier and the initial emotion classifier for classification respectively to obtain the reconstructed classification result and predicted emotion label of the converted sample audio; inputting the sample latent variables into the initial flow model for conversion to obtain the target audio; inputting the sample emotion vector into the initial emotion mapper for Perform emotion mapping to obtain predicted emotion intensity; use the initial content encoder to encode the plain text content in the sample original audio to obtain a sample content encoding vector; input the sample emotion vector and the sample content encoding vector into the initial duration predictor for duration prediction and content expansion to obtain a sample content extension sequence carrying corresponding emotions; construct a total loss function based on the reconstructed classification result, the predicted emotion label, the predicted emotion intensity, the target audio and the sample content extension sequence carrying corresponding emotions; according to the total loss function, iteratively train the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier.

[0061] Among them, the reconstructed classification results specifically include the reconstructed speech signal, and the predicted emotional labels include sadness, sadness, anger, etc.

[0062] When constructing the total loss function, a first loss function is constructed according to the target audio and the extended sequence of sample content carrying the corresponding emotion; a second loss function, a third loss function and a fourth loss function are constructed respectively according to the reconstructed classification result, the predicted emotion label and the predicted emotion intensity; the first loss function, the second loss function, the third loss function and the fourth loss function are added together to obtain the total loss function.

[0063] Specifically, the audio output by the initial flow model is used as the target audio, and the sample content extension sequence carrying the corresponding emotion output by the initial duration predictor is compared with the target audio to construct the first loss function. At the same time, the second loss function is constructed based on the reconstructed speech signal in the reconstruction classification result and the real speech signal corresponding to the original audio of the sample. The third loss function is constructed based on the predicted emotion label and the input real emotion label. The fourth loss function is constructed based on the predicted emotion intensity and the real emotion intensity. Finally, the above loss functions are added together to obtain the total loss function.

[0064] In some embodiments, in order to avoid information loss and ensure content authenticity, the embodiment of the present invention also introduces an emotion renderer in the model training process, that is, constructs an initial emotion renderer. At the same time, the sample emotion vector and the sample content coding vector are input into the initial emotion renderer for emotion rendering to obtain a sample content coding vector carrying the corresponding emotion; the sample content coding vector carrying the corresponding emotion and the sample content coding vector are input into the initial duration predictor for duration prediction and content expansion to obtain a sample content extension sequence carrying the corresponding emotion. The overall training architecture provided by the embodiment of the present invention is as follows: Figure 6 shown.

[0065] In some embodiments, in order to ensure the training accuracy of the model, the embodiments of the present invention also introduce the loss function of the speaker sample feature vector, the loss function of the Mei Er spectrogram corresponding to the converted sample audio, and the loss function of each layer of the network output in the classifier. Based on this, the method includes: constructing a fifth loss function according to the speaker sample feature vector; constructing a sixth loss function according to the Mei Er spectrogram corresponding to the converted sample audio; constructing a seventh loss function according to the output vector of each layer of the network in the initial reconstruction classifier and the initial emotion classifier; adding the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, the sixth loss function and the seventh loss function to obtain the total loss function.

[0066] The embodiment of the present invention can improve the training accuracy and convergence effect of the model by constructing a speaker sample feature vector loss function, a May-2 spectrogram loss function, and constraining the vector output by each layer of the classifier network.

[0067] Furthermore, in addition to constructing the above-mentioned loss function, the embodiment of the present invention can also construct an adversarial loss function for the sentiment classifier and the reconstruction classifier, as well as a loss function between the minimum predicted duration output by the duration predictor and the actual predicted duration, thereby further ensuring the training accuracy and convergence effect of the model.

[0068] S90. Stop iteration until the preset iteration condition is met, and output the preset identity holder, the preset emotion mapper, the preset emotion evaluator, the preset content encoder, the preset duration predictor, the preset initial flow model, the preset decoder, the preset reconstruction classifier and the preset emotion classifier.

[0069] Among them, the preset iteration condition may specifically include a preset number of iterations, and the preset number of iterations may be set according to actual business requirements, which is not specifically limited in the embodiment of the present invention.

[0070] When the preset number of iterations is reached, the iteration is stopped, and a preset identity retainer, a preset emotion mapper, a preset emotion evaluator, a preset content encoder, a preset duration predictor, a preset emotion renderer, a preset initial flow model, a preset decoder, a preset reconstruction classifier and a preset emotion classifier are output.

[0071] The embodiment of the present invention provides an emotional speech conversion method, which uses a preset identity retainer to extract the speaker feature vector corresponding to the original audio, and inputs the emotion control information into a preset emotion mapper for emotion mapping to achieve emotion control in the speech conversion process. At the same time, the plain text content in the original audio is encoded by a preset content encoder to obtain a content encoding vector, and finally, according to the speaker feature vector, the emotion vector and the content encoding vector, a converted audio corresponding to the original audio is generated. Since the embodiment of the present invention can use the preset content encoder to strip the content from the original audio when performing emotion control, and only encode the plain text content, the original emotion information can be removed from the original audio, and then combined with the emotion control information and the speaker characteristics to generate the converted audio, thereby ensuring that the converted audio is not affected by the emotion fluctuations of the original audio itself, and thus the emotion control effect can be ensured.

[0072] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0073] In one embodiment, an emotional speech conversion device is provided, which can be specifically applied to speech conversion in the financial field and the medical and elderly care field. The emotional speech conversion device corresponds to the emotional speech conversion method in the above embodiment. Figure 7 As shown, there are an acquisition unit 101, an extraction unit 102, a mapping unit 103, an encoding unit 104 and a generation unit 105.

[0074] The acquisition unit 101 may be used to acquire original audio and emotion control information.

[0075] The extraction unit 102 may be configured to extract a speaker feature vector corresponding to the original audio using a preset identity retainer.

[0076] The mapping unit 103 may be used to input the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information.

[0077] The encoding unit 104 may be configured to encode the plain text content in the original audio using a preset content encoder to obtain a content encoding vector.

[0078] The generating unit 105 may be configured to generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector.

[0079] In some embodiments, the generation unit includes: a prediction module and a generation module.

[0080] The prediction module can be used to input the emotion vector and the content encoding vector into a preset duration predictor to perform duration prediction and content expansion, so as to obtain a content expansion sequence carrying corresponding emotions.

[0081] The generation module can be used to generate converted audio corresponding to the original audio according to the content extension sequence carrying the corresponding emotion and the speaker feature vector.

[0082] In some embodiments, the prediction module can be specifically used to input the emotion vector and the content coding vector into a preset emotion renderer for emotion rendering to obtain a content coding vector carrying the corresponding emotion; input the content coding vector carrying the corresponding emotion and the content coding vector into a preset duration predictor for duration prediction and content expansion to obtain a content extension sequence carrying the corresponding emotion.

[0083] In some embodiments, the generation module can be specifically used to input the content extension sequence carrying the corresponding emotion into a preset flow model for vector conversion to obtain a target feature vector; generate latent variables based on the speaker feature vector and the target feature vector; input the latent variables into a preset decoder for decoding, and output the converted audio.

[0084] In some embodiments, the mapping unit 103 may be specifically configured to input the target emotion intensity and the target emotion label into the preset emotion mapper for emotion mapping to obtain the emotion vector.

[0085] In some embodiments, the device further includes: a construction unit, a training unit, and an output unit.

[0086] The acquisition unit 101 may also be used to acquire a sample original audio and an emotion label corresponding to the sample original audio.

[0087] The construction unit can be used to construct an initial identity retainer, an initial emotion mapper, an initial emotion evaluator, an initial content encoder, an initial duration predictor, an initial flow model, an initial decoder, an initial reconstruction classifier and an initial emotion classifier.

[0088] The training unit can be used to iteratively train the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier based on the original audio and the emotion label corresponding to the sample original audio.

[0089] The output unit can be used to stop iteration until a preset iteration condition is met, and output the preset identity holder, the preset emotion mapper, the preset emotion evaluator, the preset content encoder, the preset duration predictor, the preset initial flow model, the preset decoder, the preset reconstruction classifier and the preset emotion classifier.

[0090] In some embodiments, the training unit can be specifically used to use the initial identity retainer to extract the speaker sample feature vector corresponding to the original audio; input the sample original audio and the emotion label into the initial emotion evaluator for emotion evaluation to obtain a sample emotion vector; determine the sample latent variable according to the speaker sample feature vector and the sample emotion vector; input the sample latent variable into the initial decoder for decoding to obtain the converted sample audio; input the converted sample audio into the initial reconstruction classifier and the initial emotion classifier for classification, respectively, to obtain the reconstructed classification result and predicted emotion label of the converted sample audio; input the sample latent variable into the initial flow model for conversion to obtain the target audio; input the sample emotion vector into the initial emotion mapper for emotion Mapping to obtain predicted emotion intensity; using the initial content encoder to encode the plain text content in the sample original audio to obtain a sample content encoding vector; inputting the sample emotion vector and the sample content encoding vector into the initial duration predictor for duration prediction and content expansion to obtain a sample content extension sequence carrying corresponding emotions; constructing a total loss function according to the reconstructed classification result, the predicted emotion label, the predicted emotion intensity, the target audio and the sample content extension sequence carrying corresponding emotions; according to the total loss function, iteratively training the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier.

[0091] In some embodiments, the training unit can also be specifically used to construct a first loss function based on the target audio and the extended sequence of sample content carrying corresponding emotions; construct a second loss function, a third loss function and a fourth loss function respectively according to the reconstructed classification result, the predicted emotion label and the predicted emotion intensity; add the first loss function, the second loss function, the third loss function and the fourth loss function to obtain the total loss function.

[0092] In some embodiments, the construction unit may also be used to construct an initial emotion renderer.

[0093] The training unit can also be specifically used to input the sample emotion vector and the sample content coding vector into the initial emotion renderer for emotion rendering to obtain a sample content coding vector carrying the corresponding emotion; input the sample content coding vector carrying the corresponding emotion and the sample content coding vector into the initial duration predictor for duration prediction and content expansion to obtain a sample content extension sequence carrying the corresponding emotion.

[0094] Since the embodiment of the present invention can use a preset content encoder to strip content from the original audio and only encode the plain text content when performing emotion control, the original emotion information can be removed from the original audio, and then combined with the emotion control information and speaker characteristics to generate converted audio, thereby ensuring that the converted audio is not affected by the emotional fluctuations of the original audio itself, thereby ensuring the emotion control effect.

[0095] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps of an emotional speech conversion method.

[0096] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0097] Get raw audio and emotion control information;

[0098] Extracting a speaker feature vector corresponding to the original audio using a preset identity retainer;

[0099] Inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information;

[0100] Encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector;

[0101] Generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector.

[0102] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0103] Get raw audio and emotion control information;

[0104] Extracting a speaker feature vector corresponding to the original audio using a preset identity retainer;

[0105] Inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information;

[0106] Encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector;

[0107] Generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector.

[0108] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant description of the server side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0110] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0111] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for emotional speech conversion, characterized in that: include: Get raw audio and emotion control information; Extracting a speaker feature vector corresponding to the original audio using a preset identity retainer; Inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information; Encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector; Generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector.

2. The method according to claim 1, characterized in that: The step of generating converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content encoding vector comprises: Inputting the emotion vector and the content encoding vector into a preset duration predictor for duration prediction and content expansion to obtain a content expansion sequence carrying corresponding emotions; generating a converted audio corresponding to the original audio according to the content extension sequence carrying the corresponding emotion and the speaker feature vector; and / or When the emotion control information includes a target emotion intensity and a target emotion label, inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information includes: The target emotion intensity and the target emotion label are input into the preset emotion mapper for emotion mapping to obtain the emotion vector.

3. The method according to claim 2, characterized in that The step of inputting the emotion vector and the content coding vector into a preset duration predictor for duration prediction and content extension to obtain a content extension sequence carrying corresponding emotions includes: Inputting the emotion vector and the content coding vector into a preset emotion renderer for emotion rendering to obtain a content coding vector carrying the corresponding emotion; The content coding vector carrying the corresponding emotion and the content coding vector are input into a preset duration predictor for duration prediction and content expansion to obtain a content expansion sequence carrying the corresponding emotion.

4. The method according to claim 2, characterized in that: The step of generating converted audio corresponding to the original audio according to the content extension sequence carrying the corresponding emotion and the speaker feature vector comprises: Inputting the content extension sequence carrying the corresponding emotion into a preset flow model for vector conversion to obtain a target feature vector; Generate latent variables according to the speaker feature vector and the target feature vector; The latent variable is input into a preset decoder for decoding, and the converted audio is output.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining a sample original audio and an emotion label corresponding to the sample original audio; Constructing an initial identity preserver, an initial sentiment mapper, an initial sentiment evaluator, an initial content encoder, an initial duration predictor, an initial flow model, an initial decoder, an initial reconstruction classifier, and an initial sentiment classifier; Based on the original audio and the emotion label corresponding to the sample original audio, iteratively train the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier; Until the preset iteration condition is met, the iteration is stopped, and the preset identity retainer, the preset emotion mapper, the preset emotion evaluator, the preset content encoder, the preset duration predictor, the preset initial flow model, the preset decoder, the preset reconstruction classifier and the preset emotion classifier are output.

6. The method according to claim 5, characterized in that The method of iteratively training the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier based on the original audio and the emotion label corresponding to the sample original audio comprises: Extracting a speaker sample feature vector corresponding to the original audio using the initial identity retainer; Inputting the sample original audio and the emotion label into the initial emotion evaluator for emotion evaluation to obtain a sample emotion vector; Determining a sample latent variable according to the speaker sample feature vector and the sample emotion vector; Inputting the sample latent variable into the initial decoder for decoding to obtain converted sample audio; Inputting the converted sample audio into the initial reconstruction classifier and the initial emotion classifier for classification respectively, to obtain the reconstruction classification result and predicted emotion label of the converted sample audio; Inputting the sample latent variables into the initial flow model for conversion to obtain target audio; Inputting the sample emotion vector into the initial emotion mapper for emotion mapping to obtain predicted emotion intensity; Encoding the plain text content in the sample original audio using the initial content encoder to obtain a sample content encoding vector; Inputting the sample emotion vector and the sample content coding vector into the initial duration predictor for duration prediction and content extension to obtain a sample content extension sequence carrying corresponding emotions; Constructing a total loss function according to the reconstructed classification result, the predicted emotion label, the predicted emotion intensity, the target audio, and the extended sequence of sample content carrying the corresponding emotion; According to the total loss function, the initial identity retainer, the initial emotion mapper, the initial emotion evaluator, the initial content encoder, the initial duration predictor, the initial flow model, the initial decoder, the initial reconstruction classifier and the initial emotion classifier are iteratively trained.

7. The method according to claim 6, characterized in that The constructing a total loss function according to the reconstructed classification result, the predicted emotion label, the predicted emotion intensity, the target audio and the sample content extension sequence carrying the corresponding emotion comprises: Constructing a first loss function according to the target audio and the extended sequence of sample contents carrying corresponding emotions; According to the reconstructed classification result, the predicted emotion label and the predicted emotion intensity, respectively constructing a second loss function, a third loss function and a fourth loss function; Adding the first loss function, the second loss function, the third loss function and the fourth loss function to obtain the total loss function; and / or The method further comprises: Build the initial emotion renderer; The step of inputting the sample emotion vector and the sample content coding vector into the initial duration predictor for duration prediction and content extension to obtain a sample content extension sequence carrying corresponding emotions comprises: Inputting the sample emotion vector and the sample content coding vector into the initial emotion renderer for emotion rendering to obtain a sample content coding vector carrying the corresponding emotion; The sample content coding vector carrying the corresponding emotion and the sample content coding vector are input into the initial duration predictor for duration prediction and content extension to obtain a sample content extension sequence carrying the corresponding emotion.

8. An emotional speech conversion device, characterized in that: include: An acquisition unit, used for acquiring original audio and emotion control information; An extraction unit, configured to extract a speaker feature vector corresponding to the original audio using a preset identity retainer; A mapping unit, used for inputting the emotion control information into a preset emotion mapper for emotion mapping to obtain an emotion vector corresponding to the emotion control information; An encoding unit, used for encoding the plain text content in the original audio using a preset content encoder to obtain a content encoding vector; A generating unit is used to generate converted audio corresponding to the original audio according to the speaker feature vector, the emotion vector and the content coding vector.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the emotional speech conversion method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the emotional speech conversion method according to any one of claims 1 to 7 are implemented.

Images