Emotional speech conversion method, apparatus, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a sentiment speech conversion method, device, equipment and medium, the method comprising the following steps: extracting original audio feature representation according to input original audio waveform conversion, wherein the original audio feature representation is an original mel-frequency spectrum diagram or an SSL representation; extracting original audio HuBERT representation as a real unit sequence according to the input original audio waveform conversion; performing style reconstruction on the original audio feature representation obtained by extraction after style decomposition, combining a target style to obtain a target style coding feature vector; gathering context information by combining a target style coding feature vector according to a sentiment feature vector input by a sentiment unit through a cross-attention mechanism, and compressing the context information to a unit level; performing unit-level stylization and frame-level stylization on the context information respectively; and creating a target speech mel-frequency spectrum diagram according to a target style vector, the context information after unit-level stylization and the context information after frame-level stylization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and speech processing, and in particular to a method and device for converting emotional speech, and a related apparatus and medium. BACKGROUND

[0002] At present, emotional voice conversion (EVC) is applied in virtual assistants, Internet of Things devices, entertainment, intelligent diagnosis and treatment, remote consultation and other fields, providing users with more personalized and emotionally rich voice interaction experiences. The commonly used model for existing emotional voice conversion is a sequence-to-sequence (seq2seq) model, which can implicitly model the duration. However, using the sequence-to-sequence (seq2seq) model faces typical challenges of autoregressive models, such as long-term dependencies and repetition problems, and the voice conversion efficiency is low. SUMMARY

[0003] The present application provides a method and device for converting emotional speech, and a related apparatus and medium, to solve the technical problem of low efficiency of existing emotional voice conversion.

[0004] In a first aspect, a method for converting emotional speech is provided, comprising:

[0005] extracting original audio feature representations from input original audio waveforms, wherein the original audio feature representations are original mel-spectrograms or SSL representations;

[0006] extracting original audio HuBERT representations as real unit sequences from the input original audio waveforms;

[0007] performing style decomposition on the extracted original audio feature representations and combining target styles to perform style reconstruction, to obtain target style encoding feature vectors;

[0008] aggregating context information by a cross-attention mechanism according to emotional feature vectors input by emotional units and combining target style encoding feature vectors, and compressing to the unit level;

[0009] performing unit-level stylization and frame-level stylization on the context information, respectively;

[0010] creating a target speech mel-spectrogram according to target style vectors, unit-level stylized context information and frame-level stylized context information.

[0011] In a second aspect, a device for converting emotional speech is provided, comprising:

[0012] an original audio feature representation extraction module configured to extract original audio feature representations from input original audio waveforms, wherein the original audio feature representations are original mel-spectrograms or SSL representations;

[0013] a raw audio HuBERT representation extraction module configured to convert and extract a raw audio HuBERT representation as a real unit sequence from an input raw audio waveform;

[0014] a style decomposition and reconstruction module configured to perform style decomposition on the extracted raw audio feature representation and perform style reconstruction on the style-decomposed raw audio feature representation combined with a target style to obtain a target style encoded feature vector;

[0015] a unit-level context information conversion module configured to aggregate context information from a sentiment feature vector of a sentiment unit input combined with a target style encoded feature vector through a cross-attention mechanism and compress the context information to a unit level;

[0016] a hierarchical stylization module configured to perform unit-level stylization and frame-level stylization on the context information, respectively;

[0017] a target speech mel-spectrogram generation module configured to create a target speech mel-spectrogram from a target style vector, unit-level stylized context information, and frame-level stylized context information.

[0018] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotion speech conversion method when executing the computer program.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is executable by a processor to implement the steps of the above-mentioned emotion speech conversion method.

[0020] The scheme realized by the emotion speech conversion method, device, computer device and storage medium can receive original audio through a client, extract original audio feature representation according to input original audio waveform conversion, wherein the original audio feature representation is an original mel-frequency spectrum graph or an SSL representation; extract original audio HuBERT representation as a real unit sequence according to input original audio waveform conversion; perform style decomposition on the extracted original audio feature representation, and then perform style reconstruction in combination with a target style to obtain a target style coding feature vector; aggregate context information according to a sentiment feature vector input by a sentiment unit in combination with the target style coding feature vector through a cross-attention mechanism, and compress the context information to a unit level; perform unit-level stylization and frame-level stylization on the context information respectively; and create a target speech mel-frequency spectrum graph according to the target style vector, the context information after unit-level stylization and the context information after frame-level stylization. In the present application, emotion speech conversion can be performed in a synchronous and parallel manner using original audio features and original audio HuBERT representation, the efficiency of speech conversion is improved, synchronous language and paralanguage information is obtained through the cross-attention mechanism in combination with the sentiment feature vector input by the sentiment unit and the target style coding feature vector after style decomposition and reconstruction, the controllability of emotion language conversion is enhanced, and unit-level and frame-level stylization are performed respectively, thereby improving the quality of generated speech. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is an application environment schematic diagram of an emotion speech conversion method in an embodiment of the present application;

[0023] Figure 2 is a flowchart schematic diagram of an emotion speech conversion method in an embodiment of the present application;

[0024] Figure 3 is Figure 2 is a specific implementation flowchart schematic diagram of step S40 in the embodiment;

[0025] Figure 4 is a structure schematic diagram of an emotion speech conversion device in an embodiment of the present application;

[0026] Figure 5 is a structure schematic diagram of a computer device in an embodiment of the present application;

[0027] Figure 6This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] The emotional speech conversion method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive the input raw audio from the client and extract the raw audio feature representation based on the input raw audio waveform. The raw audio feature representation is either a raw Mel spectrogram or an SSL representation. The server then extracts the raw audio HuBERT representation as the real unit sequence based on the input raw audio waveform. After style decomposition of the extracted raw audio feature representation, style reconstruction is performed by combining it with the target style to obtain the target style encoding feature vector. A cross-attention mechanism is used to aggregate contextual information based on the emotional feature vector input from the emotional unit and the target style encoding feature vector, compressing the contextual information to the unit level. Unit-level and frame-level stylization are performed on the contextual information. A target speech Mel spectrogram is created based on the target style vector, the unit-level stylized contextual information, and the frame-level stylized contextual information. In this invention, emotional speech conversion can be performed synchronously and in parallel using raw audio features and raw audio HuBERT representation, improving the efficiency of speech conversion. The cross-attention mechanism combines the emotional feature vector input from the emotional unit and the target style encoding feature vector reconstructed by style decomposition to synchronize linguistic and paralinguistic information, enhancing the controllability of emotional speech conversion. Furthermore, unit-level and frame-level stylization are performed separately, improving the quality of the generated speech. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0030] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the emotional speech conversion method provided in this embodiment of the invention includes the following steps:

[0031] S10: Extract the original audio feature representation based on the input original audio waveform. The original audio feature representation is either the original Mel spectrogram or the SSL representation. SSL (Self-supervised Learning) refers to a machine learning method that directly mines self-supervised information from large-scale unsupervised data for supervised learning and training.

[0032] S20: Extract the original audio HuBERT representation as the ground truth unit sequence based on the input original audio waveform; the original audio HuBERT representation is obtained by using the HuBERT model as a feature extractor to extract features from the input original audio waveform. The HuBERT model is a pre-trained unsupervised speech representation model that uses a Transformer structure model to predict the target label of the masked frame by inputting a randomly masked speech feature sequence, thereby obtaining a high-quality speech feature representation.

[0033] S30: After performing style decomposition on the extracted original audio feature representation, style reconstruction is performed by combining it with the target style to obtain the target style encoding feature vector;

[0034] In step S30, which involves performing style decomposition on the extracted original audio feature representation and then combining it with the target style for style reconstruction to obtain the target style encoding feature vector, the following steps are included:

[0035] The style autoencoder uses a destylating Transformer to perform style decomposition on the extracted original audio feature representation, obtaining style-independent audio features and destylated audio feature representation; specifically, the destylating Transformer separates style-independent audio features from the original audio features.

[0036] A stylized Transformer from a style autoencoder is used to reconstruct the style of the destylated audio feature representation, combining the target style with the stylized style, to obtain the target style encoded feature vector. Specifically, the stylized Transformer applies the target style to the destylated audio feature representation for style reconstruction.

[0037] S40: By using a cross-attention mechanism, contextual information is gathered based on the emotional feature vector input from the emotional unit and the target style encoding feature vector, and then compressed to the unit level.

[0038] Among them, such as Figure 3 As shown, in step S40, which involves using a cross-attention mechanism to aggregate contextual information based on the sentiment feature vector input to the sentiment unit and combining it with the target style encoding feature vector, and compressing it to the unit level, the following steps are included:

[0039] S41: Adopting a unit aligner to calculate an attention weight matrix according to the obtained target style encoding feature vector and the emotion feature vector transmitted by the emotion unit through a cross attention mechanism; wherein the emotion feature vector transmitted by the emotion unit is the converted emotion feature vector of the target emotion.

[0040] S42: Calculate a cross entropy loss according to the calculated attention weight matrix combined with the real unit sequence, and constrain the values in the attention matrix through the calculated cross entropy loss, so that the values in the attention matrix through the calculated cross entropy loss are as consistent as possible with the content in the original audio.

[0041] S43: According to the probability value in the calculated attention weight matrix, predict the predicted HuBERT representation as a predicted unit sequence.

[0042] S44: De-duplicate the predicted unit sequence, extract the unit value and the unit duration, wherein the unit duration refers to the number of consecutive occurrences of the same unit value.

[0043] Specifically, after step S44, it further includes:

[0044] Pool the obtained unit value and unit duration, and input to the hierarchical stylization encoder.

[0045] Specifically, after step S44, it further includes:

[0046] According to the unit duration as the target duration, train the duration predictor. Wherein, by training the duration predictor, the mapping relationship between the input attention weight matrix and the output duration sequence can be constructed by the duration predictor according to the real duration, and the voice duration can be flexibly controlled, so that the generated emotional voice is more natural.

[0047] S50: Perform unit-level stylization and frame-level stylization on the context information, respectively;

[0048] In step S50, that is, performing unit-level stylization and frame-level stylization on the context information, respectively, includes the following steps:

[0049] The unit-level stylization Transformer of the hierarchical stylization encoder processes the context information compressed to the unit level;

[0050] The length adjuster expands the context information compressed to the unit level to the frame level to obtain the frame-level context information;

[0051] The frame-level stylization Transformer of the hierarchical stylization encoder processes the obtained frame-level context information.

[0052] S60: creating a target speech mel-spectrogram according to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization.

[0053] In step S60, specifically:

[0054] The diffusion model of the diffusion-based generator is guided to generate the target speech mel-spectrogram by taking the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization as conditions.

[0055] Specifically, after step S60, the method further includes:

[0056] The obtained target speech mel-spectrogram is converted by using a vocoder to obtain a target emotion conversion speech waveform, and the target emotion conversion speech audio is directly outputted, so that the server returns the emotion conversion target emotion speech audio to the client.

[0057] It can be seen that in the above scheme, the emotion speech conversion can be performed in a synchronous and parallel manner using the original audio features and the original audio HuBERT representation, the efficiency of the speech conversion is improved, the cross-attention mechanism is combined with the emotion feature vector input by the emotion unit and the target style code feature vector after style decomposition and reconstruction to synchronize the language and paralanguage information, the controllability of the emotion language conversion is enhanced, and the unit-level and frame-level stylization are performed respectively, the quality of the generated speech is improved.

[0058] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0059] In an embodiment, an emotion speech conversion device is provided, which corresponds to the emotion speech conversion method in the above embodiments. As shown in the figure, the emotion speech conversion device includes an original audio feature representation extraction module 101, an original audio HuBERT representation extraction module 102, a style decomposition and reconstruction module 103, a unit-level context information conversion module 104, a hierarchical stylization module 105, and a target speech mel-spectrum generation module 106. The functions of each module are described in detail as follows: Figure 4

[0060] The original audio feature representation extraction module 101 is configured to extract the original audio feature representation from the input original audio waveform, wherein the original audio feature representation is an original mel-spectrogram or an SSL representation.

[0061] The original audio HuBERT representation extraction module 102 is configured to extract the original audio HuBERT representation as a real unit sequence from the input original audio waveform.​

[0062] The style decomposition and reconstruction module 103 is configured to perform style decomposition on the extracted original audio feature representation and perform style reconstruction on the original audio feature representation combined with a target style to obtain a target style encoded feature vector.

[0063] The unit-level context information conversion module 104 is configured to aggregate context information according to the emotion feature vector input by the emotion unit and the target style encoded feature vector by a cross-attention mechanism and compress the context information to a unit level.

[0064] The hierarchical stylization module 105 is configured to perform unit-level stylization and frame-level stylization on the context information, respectively.

[0065] The target speech mel-spectrogram generation module 106 is configured to create a target speech mel-spectrogram according to the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

[0066] In an embodiment, the style decomposition and reconstruction module 103 is specifically configured to:

[0067] The style decomposition and reconstruction module 103 is configured to perform style decomposition on the extracted original audio feature representation and perform style reconstruction on the original audio feature representation combined with a target style to obtain a target style encoded feature vector.

[0068] The style decomposition and reconstruction module 103 is configured to perform style decomposition on the extracted original audio feature representation and perform style reconstruction on the original audio feature representation combined with a target style to obtain a target style encoded feature vector.

[0069] In an embodiment, the unit-level context information conversion module 104 is specifically configured to:

[0070] The unit-level context information conversion module 104 is configured to calculate an attention weight matrix according to the obtained target style encoded feature vector and the emotion feature vector transmitted by the emotion unit by a cross-attention mechanism.

[0071] The unit-level context information conversion module 104 is configured to calculate a cross-entropy loss according to the calculated attention weight matrix and the real unit sequence and constrain the values in the attention matrix by the calculated cross-entropy loss.

[0072] The unit-level context information conversion module 104 is configured to predict a predicted HuBERT representation as a predicted unit sequence according to the probability values in the calculated attention weight matrix.

[0073] The unit-level context information conversion module 104 is configured to remove duplicates from the predicted unit sequence and extract unit values and unit durations.

[0074] In an embodiment, the unit-level context information conversion module 104 is further configured to:

[0075] The length predictor is trained according to the unit length as the target length.

[0076] In an embodiment, the hierarchical stylization module 105 is specifically configured to:

[0077] The unit-level stylization Transformer adopting the hierarchical stylization encoder processes the context information compressed to the unit level;

[0078] The length adjuster is adopted to expand the context information compressed to the unit level to the frame level to obtain the context information of the frame level;

[0079] The frame-level stylization Transformer adopting the hierarchical stylization encoder processes the obtained context information of the frame level.

[0080] In an embodiment, the target speech mel spectrum generation module 106 is specifically configured to:

[0081] The diffusion model of the diffusion-based generator is guided to generate a target speech mel spectrum graph by taking the target style vector, the context information after unit-level stylization and the context information after frame-level stylization as conditions.

[0082] In an embodiment, the target speech mel spectrum generation module 106 is further configured to:

[0083] The vocoder is adopted to convert the obtained target speech mel spectrum graph to obtain a target emotion conversion speech waveform.

[0084] The present application provides an emotion speech conversion device, which converts emotion speech by using original audio features and original audio HuBERT representation in a synchronous and parallel manner, improves the efficiency of speech conversion, synchronizes language and paralanguage information by combining emotion feature vectors input by emotion units and target style coding feature vectors reconstructed by style decomposition through a cross-attention mechanism, enhances the controllability of emotion language conversion, and performs unit-level and frame-level stylization respectively, thereby improving the quality of generated speech.

[0085] The specific limitations of the emotion speech conversion device can be referred to the limitations of the emotion speech conversion method in the above, which will not be repeated here. Each module in the above emotion speech conversion device can be realized by software, hardware and their combinations in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0086] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the function or step of the emotion speech conversion method server side.

[0087] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 6 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the function or step of the emotion speech conversion method client side

[0088] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the following steps:

[0089] Converting and extracting the original audio feature representation according to the input original audio waveform, wherein the original audio feature representation is an original mel-spectrogram or an SSL representation;

[0090] Converting and extracting the original audio HuBERT representation as a real unit sequence according to the input original audio waveform;

[0091] After style decomposition is performed on the original audio feature representation obtained by extraction, the target style is combined for style reconstruction to obtain a target style code feature vector;

[0092] Through a cross-attention mechanism, context information is aggregated according to the emotion feature vector input by the emotion unit combined with the target style code feature vector, and is compressed to the unit level;

[0093] The context information is respectively subjected to unit-level stylization and frame-level stylization;

[0094] According to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization, a target speech mel spectrum graph is created.

[0095] In one embodiment, a computer readable storage medium is provided, and the computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps:

[0096] According to the input original audio waveform, an original audio feature representation is converted and extracted, wherein the original audio feature representation is an original mel spectrum graph or an SSL representation;

[0097] According to the input original audio waveform, an original audio HuBERT representation is converted and extracted as a real unit sequence;

[0098] The original audio feature representation obtained by extraction is subjected to style decomposition and combined with the target style for style reconstruction to obtain a target style coding feature vector;

[0099] Through a cross-attention mechanism, context information is aggregated according to the emotion feature vector input by the emotion unit and combined with the target style coding feature vector, and is compressed to a unit level;

[0100] The context information is respectively subjected to unit-level stylization and frame-level stylization;

[0101] According to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization, a target speech mel spectrum graph is created.

[0102] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement are described above with reference to the related descriptions of the server side and the client side in the foregoing method embodiments, and to avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0105] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An emotional speech conversion method, characterized by, The method comprises the following steps: extracting a raw audio feature representation from an input raw audio waveform, wherein the raw audio feature representation is a raw mel-spectrogram or an SSL representation; extracting a raw audio HuBERT representation as a real unit sequence from the input raw audio waveform; performing style decomposition on the extracted raw audio feature representation and combining a target style to perform style reconstruction, to obtain a target style encoding feature vector; aggregating context information by a cross-attention mechanism according to a sentiment feature vector input by a sentiment unit and combining the target style encoding feature vector, and compressing the context information to a unit level; performing unit-level stylization and frame-level stylization on the context information, respectively; creating a target speech mel-spectrogram according to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization.

2. The emotional speech conversion method of claim 1, wherein, The method further comprises the following steps: performing style decomposition on the extracted raw audio feature representation by a style-agnostic Transformer of a style auto-encoder, to obtain a style-independent audio feature and a de-stylized audio feature representation; performing style reconstruction on the de-stylized audio feature representation by a stylization Transformer of the style auto-encoder in combination with the target style, to obtain the target style encoding feature vector.

3. The emotional speech conversion method of claim 1, wherein, The method further comprises the following steps: calculating an attention weight matrix by a unit aligner according to the obtained target style encoding feature vector and the sentiment feature vector transmitted by the sentiment unit through the cross-attention mechanism; calculating a cross-entropy loss according to the calculated attention weight matrix and the real unit sequence, and constraining the values in the attention matrix by the calculated cross-entropy loss; predicting a predicted HuBERT representation as a predicted unit sequence according to the probability values in the calculated attention weight matrix; performing deduplication on the predicted unit sequence, to extract unit values and unit durations.

4. The emotional speech conversion method of claim 3, wherein, After the step of performing deduplication on the predicted unit sequence, to extract unit values and unit durations, the method further comprises the following steps: training a duration predictor according to the unit durations as target durations.

5. The emotional speech conversion method of claim 1, wherein, The method further comprises the following steps: processing the context information compressed to the unit level by a unit-level stylization Transformer of a hierarchical stylization encoder; expanding the context information compressed to the unit level to the frame level by a length adjuster, to obtain frame-level context information; processing the obtained frame-level context information by a frame-level stylization Transformer of the hierarchical stylization encoder.

6. The emotional speech conversion method of claim 1, wherein, The step of creating a target speech mel-spectrogram according to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization is specifically performed as follows: The diffusion model of the diffusion-based generator is guided to generate a target speech mel spectrum graph based on the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization.

7. The emotional voice conversion method of claim 1, wherein, After the target speech mel spectrum graph is created according to the target style vector, the context information after unit-level stylization, and the context information after frame-level stylization, the method further includes: The generated target speech mel spectrum graph is converted by using a vocoder to obtain a target emotion conversion speech waveform.

8. An emotional speech conversion apparatus characterized by comprising: Comprise: An original audio feature representation extraction module is configured to extract an original audio feature representation from an input original audio waveform, wherein the original audio feature representation is an original mel spectrum graph or an SSL representation; An original audio HuBERT representation extraction module is configured to extract an original audio HuBERT representation as a real unit sequence from an input original audio waveform; A style decomposition and reconstruction module is configured to perform style decomposition on the extracted original audio feature representation and combine target styles to perform style reconstruction, thereby obtaining a target style encoding feature vector; A unit-level context information conversion module is configured to aggregate context information by using a cross-attention mechanism according to emotion features of an emotion unit and combining a target style encoding feature vector, and compress the context information to a unit level; A hierarchical stylization module is configured to perform unit-level stylization and frame-level stylization on context information, respectively; A target speech mel spectrum generation module is configured to create a target speech mel spectrum graph according to a target style vector, context information after unit-level stylization, and context information after frame-level stylization.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the emotion speech conversion method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the emotion speech conversion method according to any one of claims 1 to 7.

Images