Emotional voice conversion method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an emotional speech conversion method, device, equipment and medium, and the method comprises the steps: converting and extracting an original audio feature representation according to an input original audio waveform, the original audio feature representation being an original Mel spectrogram or SSL representation; converting and extracting an original audio HuBERT representation as a real unit sequence according to an input original audio waveform; performing style decomposition on the extracted original audio feature representation, and performing style reconstruction in combination with a target style to obtain a target style coding feature vector; contextual information is gathered through a cross attention mechanism according to an emotion feature vector input by an emotion unit in combination with a target style coding feature vector, and the contextual information is compressed to a unit level; respectively carrying out unit-level stylization and frame-level stylization on the context information; and according to the target style vector, the context information after the unit-level stylization and the context information after the frame-level stylization, creating a target voice Mel spectrogram.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and speech processing technology, and in particular to an emotional speech conversion method, device, equipment and medium. Background Art

[0002] At present, emotional voice conversion (EVC) is applied in virtual assistants, IoT devices, entertainment, intelligent diagnosis and treatment, remote consultation and other fields, providing users with a more humane and emotionally rich voice interaction experience. The existing model commonly used for emotional voice conversion is the sequence-to-sequence (seq2seq) model, which can implicitly model duration. However, the use of the sequence-to-sequence (seq2seq) model faces typical challenges of autoregressive models, such as long-term dependency and repetition problems, and low voice conversion efficiency. Summary of the invention

[0003] The present invention provides an emotional speech conversion method, device, equipment and medium to solve the technical problem of low efficiency of existing emotional speech conversion.

[0004] In a first aspect, a method for emotional speech conversion is provided, comprising:

[0005] Extracting original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrogram or SSL representation;

[0006] Extract the original audio HuBERT representation as a true unit sequence based on the input original audio waveform conversion;

[0007] After performing style decomposition on the extracted original audio feature representation, style reconstruction is performed in combination with the target style to obtain the target style encoding feature vector;

[0008] Through the cross-attention mechanism, the context information is aggregated based on the emotional feature vector input by the emotional unit and the target style encoding feature vector, and then compressed to the unit level;

[0009] Perform unit-level stylization and frame-level stylization on context information respectively;

[0010] Create a target speech mel-spectrogram based on the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

[0011] In a second aspect, an emotional speech conversion device is provided, comprising:

[0012] An original audio feature representation extraction module is used to extract the original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrum map or SSL representation;

[0013] The original audio HuBERT representation extraction module is used to extract the original audio HuBERT representation as a real unit sequence according to the input original audio waveform conversion;

[0014] The style decomposition and reconstruction module is used to perform style decomposition on the extracted original audio feature representation and then reconstruct the style based on the target style to obtain the target style encoding feature vector;

[0015] The unit-level context information conversion module is used to aggregate context information based on the emotional feature vector input by the emotional unit and the target style encoding feature vector through a cross-attention mechanism, and compress it to the unit level;

[0016] Hierarchical stylization module, which is used to perform unit-level stylization and frame-level stylization on context information respectively;

[0017] The target speech mel-spectrogram generation module is used to create a target speech mel-spectrogram according to the target style vector, the unit-level stylized context information and the frame-level stylized context information.

[0018] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotional speech conversion method when executing the computer program.

[0019] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned emotional speech conversion method are implemented.

[0020] In the scheme implemented by the above-mentioned emotional speech conversion method, device, computer equipment and storage medium, the original audio can be received through the client, and the original audio feature representation can be extracted according to the input original audio waveform conversion, wherein the original audio feature representation is the original Mel spectrum map or SSL representation; the original audio HuBERT representation is extracted as a real unit sequence according to the input original audio waveform conversion; the extracted original audio feature representation is style decomposed and then combined with the target style to reconstruct the style to obtain the target style encoding feature vector; the context information is aggregated according to the emotional feature vector input by the emotional unit and the target style encoding feature vector through the cross attention mechanism, and The method comprises the following steps: compressing the audio signal to the unit level; stylizing the context information at the unit level and stylizing the context information at the frame level respectively; creating a target speech mel-spectrogram according to the target style vector, the context information after unit level stylization and the context information after frame level stylization. In the present invention, the original audio features and the original audio HuBERT representation can be used in a synchronous and parallel manner to perform emotional speech conversion, thereby improving the efficiency of speech conversion. The cross-attention mechanism is combined with the emotional feature vector input by the emotional unit and the target style encoding feature vector after style decomposition and reconstruction to synchronize language and paralanguage information, thereby enhancing the controllability of emotional language conversion, and performing unit level and frame level stylization respectively to improve the quality of generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a schematic diagram of an application environment of an emotional speech conversion method in one embodiment of the present invention;

[0023] Figure 2 is a flow chart of an emotional speech conversion method according to an embodiment of the present invention;

[0024] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S40;

[0025] Figure 4 is a structural schematic diagram of an emotional speech conversion device in one embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0027] Figure 6It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The emotional speech conversion method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, a client communicates with a server through a network. The server can receive the input raw audio through the client, and extract the raw audio feature representation according to the input raw audio waveform conversion, wherein the raw audio feature representation is an original Mel spectrum map or an SSL representation; extract the raw audio HuBERT representation as a real unit sequence according to the input raw audio waveform conversion; perform style decomposition on the extracted raw audio feature representation and then perform style reconstruction in combination with the target style to obtain a target style encoding feature vector; gather context information according to the emotional feature vector input by the emotional unit and the target style encoding feature vector through a cross-attention mechanism, and compress the context information to the unit level; perform unit-level stylization and frame-level stylization on the context information respectively; create a target speech Mel spectrum map according to the target style vector, the context information after unit-level stylization and the context information after frame-level stylization. In the present invention, the original audio features and the original audio HuBERT representation can be used in a synchronous and parallel manner to perform emotional speech conversion, thereby improving the efficiency of speech conversion. The emotional feature vector input by the emotional unit and the target style encoding feature vector after style decomposition and reconstruction are combined through a cross-attention mechanism to synchronize language and paralanguage information, thereby enhancing the controllability of emotional language conversion, and performing unit-level and frame-level stylization respectively to improve the quality of generated speech. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 As shown, Figure 2 A flow chart of the emotional speech conversion method provided by an embodiment of the present invention includes the following steps:

[0031] S10: extracting original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel-spectrogram or SSL representation. SSL (Self-supervised Learning) representation refers to a machine learning method that directly mines self-supervisory information from large-scale unsupervised data for supervised learning and training.

[0032] S20: Extract the original audio HuBERT representation as a real unit sequence according to the input original audio waveform conversion; wherein the original audio HuBERT representation is obtained by using the HuBERT model as a feature extractor to extract features from the input original audio waveform. The HuBERT model is a pre-trained unsupervised speech representation model, which obtains high-quality speech feature representation by inputting a randomly masked speech feature sequence and predicting the target label of the masked frame using the Transformer structure model.

[0033] S30: performing style decomposition on the extracted original audio feature representation and then reconstructing the style in combination with the target style to obtain a target style encoding feature vector;

[0034] Among them, step S30, that is, performing style decomposition on the extracted original audio feature representation and then reconstructing the style in combination with the target style to obtain the target style encoding feature vector, includes the following steps:

[0035] A de-stylized Transformer of a style autoencoder is used to perform style decomposition on the extracted original audio feature representation to obtain audio features independent of style and de-stylized audio feature representation; wherein, the de-stylized Transformer is used to separate audio features independent of style from the original audio features.

[0036] The style transformer of the style autoencoder is used to reconstruct the style of the de-stylized audio feature representation in combination with the target style to obtain the target style encoding feature vector. The style transformer is used to apply the target style to the de-stylized audio feature representation to reconstruct the style.

[0037] S40: Through the cross-attention mechanism, the contextual information is aggregated based on the sentiment feature vector input by the sentiment unit and the target style encoding feature vector, and compressed to the unit level.

[0038] Among them, Figure 3 As shown, in step S40, context information is aggregated according to the emotional feature vector input by the emotional unit and the target style encoding feature vector through the cross attention mechanism, and compressed to the unit level, including the following steps:

[0039] S41: The unit aligner is used to calculate the attention weight matrix through the cross-attention mechanism according to the obtained target style encoding feature vector and the emotional feature vector transmitted by the emotional unit; wherein the emotional feature vector transmitted by the emotional unit is the emotional feature vector of the converted target emotion.

[0040] S42: Calculate the cross entropy loss based on the calculated attention weight matrix combined with the real unit sequence, and constrain the values ​​in the attention matrix through the calculated cross entropy loss so that the values ​​in the attention matrix are as consistent as possible with the content in the original audio.

[0041] S43: Obtain the predicted HuBERT representation as a prediction unit sequence based on the probability value prediction in the calculated attention weight matrix.

[0042] S44: De-duplicate the prediction unit sequence, extract the unit value and unit duration, wherein the unit duration refers to the number of consecutive occurrences of the same unit value.

[0043] Specifically, after step S44, the method further includes:

[0044] The obtained unit values ​​and unit durations are pooled and input into the hierarchical stylization encoder.

[0045] Specifically, after step S44, the method further includes:

[0046] The duration predictor is trained based on the unit duration as the target duration. By training the duration predictor, the duration predictor can be used to construct a mapping relationship between the input attention weight matrix and the output duration sequence based on the actual duration, so that the speech duration can be flexibly controlled, making the generated emotional speech more natural.

[0047] S50: performing unit-level stylization and frame-level stylization on the context information respectively;

[0048] Wherein, in step S50, that is, performing unit-level stylization and frame-level stylization on the context information respectively, the following steps are included:

[0049] The unit-level stylized Transformer with a hierarchical stylized encoder processes the context information compressed to the unit level;

[0050] A length adjuster is used to expand and refine the context information compressed to the unit level to the frame level to obtain the frame level context information;

[0051] The frame-level stylized Transformer of the hierarchical stylized encoder is used to process the frame-level contextual information.

[0052] S60: Create a target speech mel-spectrogram according to the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

[0053] Wherein, step S60 is specifically:

[0054] The target style vector, the unit-level stylized context information and the frame-level stylized context information are used as conditions to guide the diffusion model of the diffusion-based generator to generate the target speech mel-spectrogram.

[0055] Specifically, after step S60, the following steps are further included:

[0056] The generated target speech mel-spectrogram is converted by a vocoder to obtain a target emotion-converted speech waveform, so as to directly output the target emotion-converted speech audio, so that the server returns the target emotion-converted speech audio to the client.

[0057] It can be seen that in the above scheme, the original audio features and the original audio HuBERT representation can be used in a synchronous and parallel manner to perform emotional speech conversion, thereby improving the efficiency of speech conversion. The cross-attention mechanism is combined with the emotional feature vector input by the emotional unit and the target style encoding feature vector after style decomposition and reconstruction to synchronize language and paralanguage information, thereby enhancing the controllability of emotional language conversion, and performing unit-level and frame-level stylization respectively to improve the quality of generated speech.

[0058] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0059] In one embodiment, an emotional speech conversion device is provided, and the emotional speech conversion device corresponds to the emotional speech conversion method in the above embodiment. Figure 4 As shown, the emotional speech conversion device includes an original audio feature representation extraction module 101, an original audio HuBERT representation extraction module 102, a style decomposition and reconstruction module 103, a unit-level context information conversion module 104, a hierarchical stylization module 105 and a target speech Mel spectrum generation module 106. The functional modules are described in detail as follows:

[0060] The original audio feature representation extraction module 101 is used to extract the original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel-spectrogram or SSL representation;

[0061] The original audio HuBERT representation extraction module 102 is used to extract the original audio HuBERT representation as a real unit sequence according to the input original audio waveform conversion;

[0062] The style decomposition and reconstruction module 103 is used to perform style decomposition on the extracted original audio feature representation and then perform style reconstruction in combination with the target style to obtain a target style encoding feature vector;

[0063] The unit-level context information conversion module 104 is used to aggregate the context information according to the emotion feature vector input by the emotion unit and the target style encoding feature vector through a cross attention mechanism, and compress it to the unit level;

[0064] A hierarchical stylization module 105, for performing unit-level stylization and frame-level stylization on context information;

[0065] The target speech mel-spectrogram generation module 106 is used to create a target speech mel-spectrogram according to the target style vector, the unit-level stylized context information and the frame-level stylized context information.

[0066] In one embodiment, the style decomposition and reconstruction module 103 is specifically used to:

[0067] The de-stylized Transformer of the style autoencoder is used to perform style decomposition on the extracted original audio feature representation to obtain style-independent audio features and de-stylized audio feature representations.

[0068] Combined with the target style, the style transformer of the style autoencoder is used to reconstruct the style of the de-stylized audio feature representation to obtain the target style encoding feature vector.

[0069] In one embodiment, the unit-level context information conversion module 104 is specifically configured to:

[0070] The unit aligner is used to calculate the attention weight matrix based on the obtained target style encoding feature vector and the emotional feature vector transmitted by the emotional unit through the cross attention mechanism;

[0071] The cross entropy loss is calculated based on the calculated attention weight matrix combined with the real unit sequence, and the value in the attention matrix is ​​constrained by the calculated cross entropy loss;

[0072] Obtain the predicted HuBERT representation as a sequence of prediction units based on the probability values ​​in the calculated attention weight matrix;

[0073] De-duplicate the prediction unit sequence and extract the unit value and unit duration.

[0074] In one embodiment, the unit-level context information conversion module 104 is further configured to:

[0075] The duration predictor is trained based on the unit duration as the target duration.

[0076] In one embodiment, the hierarchical stylization module 105 is specifically used for:

[0077] The unit-level stylized Transformer with a hierarchical stylized encoder processes the context information compressed to the unit level;

[0078] A length adjuster is used to expand and refine the context information compressed to the unit level to the frame level to obtain the frame level context information;

[0079] The frame-level stylized Transformer of the hierarchical stylized encoder is used to process the frame-level contextual information.

[0080] In one embodiment, the target speech mel spectrum generating module 106 is specifically used for:

[0081] The target style vector, the unit-level stylized context information and the frame-level stylized context information are used as conditions to guide the diffusion model of the diffusion-based generator to generate the target speech mel-spectrogram.

[0082] In one embodiment, the target speech mel spectrum generating module 106 is further configured to:

[0083] The generated target speech mel-spectrogram is converted by a vocoder to obtain the target emotion-converted speech waveform.

[0084] The present invention provides an emotional speech conversion device, which performs emotional speech conversion by using original audio features and original audio HuBERT representation in a synchronous and parallel manner to improve the efficiency of speech conversion, and synchronizes language and paralanguage information through a cross-attention mechanism combined with an emotional feature vector input by an emotional unit and a target style encoding feature vector after style decomposition and reconstruction, thereby enhancing the controllability of emotional language conversion, and performing unit-level and frame-level stylization respectively to improve the quality of generated speech.

[0085] For the specific definition of the emotional speech conversion device, please refer to the definition of the emotional speech conversion method above, which will not be repeated here. The various modules in the above-mentioned emotional speech conversion device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0086] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an emotional speech conversion method.

[0087] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of an emotional speech conversion method

[0088] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0089] Extracting original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrogram or SSL representation;

[0090] Extract the original audio HuBERT representation as a true unit sequence based on the input original audio waveform conversion;

[0091] After performing style decomposition on the extracted original audio feature representation, style reconstruction is performed in combination with the target style to obtain the target style encoding feature vector;

[0092] Through the cross-attention mechanism, the context information is aggregated based on the emotional feature vector input by the emotional unit and the target style encoding feature vector, and then compressed to the unit level;

[0093] Perform unit-level stylization and frame-level stylization on context information respectively;

[0094] Create a target speech mel-spectrogram based on the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

[0095] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0096] Extracting original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrogram or SSL representation;

[0097] Extract the original audio HuBERT representation as a true unit sequence based on the input original audio waveform conversion;

[0098] After performing style decomposition on the extracted original audio feature representation, style reconstruction is performed in combination with the target style to obtain the target style encoding feature vector;

[0099] Through the cross-attention mechanism, the context information is aggregated based on the emotional feature vector input by the emotional unit and the target style encoding feature vector, and then compressed to the unit level;

[0100] Perform unit-level stylization and frame-level stylization on context information respectively;

[0101] Create a target speech mel-spectrogram based on the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

[0102] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0103] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0104] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0105] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for emotional speech conversion, characterized in that: include: Extracting original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrogram or SSL representation; Extract the original audio HuBERT representation as a true unit sequence based on the input original audio waveform conversion; After performing style decomposition on the extracted original audio feature representation, style reconstruction is performed in combination with the target style to obtain the target style encoding feature vector; Through the cross-attention mechanism, the context information is aggregated based on the emotional feature vector input by the emotional unit and the target style encoding feature vector, and then compressed to the unit level; Perform unit-level stylization and frame-level stylization on context information respectively; Create a target speech mel-spectrogram based on the target style vector, the unit-level stylized context information, and the frame-level stylized context information.

2. The emotional speech conversion method according to claim 1, wherein: The step of performing style decomposition on the extracted original audio feature representation and then reconstructing the style in combination with the target style to obtain the target style encoding feature vector includes: The de-stylized Transformer of the style autoencoder is used to perform style decomposition on the extracted original audio feature representation to obtain style-independent audio features and de-stylized audio feature representations. Combined with the target style, the style transformer of the style autoencoder is used to reconstruct the style of the de-stylized audio feature representation to obtain the target style encoding feature vector.

3. The emotional speech conversion method according to claim 1, wherein: The cross-attention mechanism aggregates context information based on the emotional feature vector input by the emotional unit and the target style encoding feature vector, and compresses it to the unit level, including: The unit aligner is used to calculate the attention weight matrix based on the obtained target style encoding feature vector and the emotional feature vector transmitted by the emotional unit through the cross attention mechanism; The cross entropy loss is calculated based on the calculated attention weight matrix combined with the real unit sequence, and the value in the attention matrix is ​​constrained by the calculated cross entropy loss; Obtain the predicted HuBERT representation as a sequence of prediction units based on the probability values ​​in the calculated attention weight matrix; De-duplicate the prediction unit sequence and extract the unit value and unit duration.

4. The emotional speech conversion method as claimed in claim 3, characterized in that: After removing duplicates from the prediction unit sequence and extracting unit values ​​and unit durations, the method further includes: The duration predictor is trained based on the unit duration as the target duration.

5. The emotional speech conversion method according to claim 1, wherein: The performing unit-level stylization and frame-level stylization on the context information respectively includes: The unit-level stylized Transformer with a hierarchical stylized encoder processes the context information compressed to the unit level; A length adjuster is used to expand and refine the context information compressed to the unit level to the frame level to obtain the frame level context information; The frame-level stylized Transformer of the hierarchical stylized encoder is used to obtain the frame-level contextual information.

6. The emotional speech conversion method according to claim 1, characterized in that: The target speech mel-spectrogram is created according to the target style vector, the unit-level stylized context information and the frame-level stylized context information, specifically: The target style vector, the unit-level stylized context information and the frame-level stylized context information are used as conditions to guide the diffusion model of the diffusion-based generator to generate the target speech mel-spectrogram.

7. The emotional speech conversion method according to claim 1, characterized in that: After creating the target speech mel-spectrogram according to the target style vector, the unit-level stylized context information and the frame-level stylized context information, the method further includes: The generated target speech mel-spectrogram is converted by a vocoder to obtain the target emotion-converted speech waveform.

8. An emotional speech conversion device, characterized in that: include: An original audio feature representation extraction module is used to extract the original audio feature representation according to the input original audio waveform conversion, wherein the original audio feature representation is an original Mel spectrum map or SSL representation; The original audio HuBERT representation extraction module is used to extract the original audio HuBERT representation as a real unit sequence according to the input original audio waveform conversion; The style decomposition and reconstruction module is used to perform style decomposition on the extracted original audio feature representation and then reconstruct the style based on the target style to obtain the target style encoding feature vector; The unit-level context information conversion module is used to aggregate context information based on the emotional feature vector input by the emotional unit and the target style encoding feature vector through a cross-attention mechanism, and compress it to the unit level; Hierarchical stylization module, which is used to perform unit-level stylization and frame-level stylization on context information respectively; The target speech mel-spectrogram generation module is used to create a target speech mel-spectrogram according to the target style vector, the unit-level stylized context information and the frame-level stylized context information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the emotional speech conversion method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the emotional speech conversion method according to any one of claims 1 to 7 are implemented.

Images